[Feature][chunjun-core] Supports capturing dirty data from the source and when the source sends it downstream #1901#1902
Merged
zoudaokoulife merged 1 commit intoDTStack:masterfrom Jul 10, 2024
Conversation
… and when the source sends it downstream DTStack#1901
Contributor
Author
|
问题再现: CREATE` TABLE source
(
`ID` int,
`FloatColumn` string,
`BinaryColumn` bytes,
`VarBinaryColumn` bytes,
`LongBinaryColumn` bytes
) WITH (
'connector' = 'xxx-x',
);
CREATE TABLE sink
(
`ID` int,
`FloatColumn` int,
`BinaryColumn` string,
`VarBinaryColumn` string,
`LongBinaryColumn` string
) WITH (
'connector' = 'stream-x'
);
insert into sink
select
`ID` as `ID`,
CAST(`FloatColumn` AS int) as `FloatColumn`, --比如这里数据源来一条脏数据为:111aa, 数据发送到下游算子时会报错,但此时脏数据无法捕获,脏数据管理器的能力就发挥不出来
CAST(`BinaryColumn` AS string) as `BinaryColumn`,
CAST(`VarBinaryColumn` AS string) as `VarBinaryColumn`,
CAST(`LongBinaryColumn` AS string) as `LongBinaryColumn`
from source ; |
Contributor
Author
|
本featrue主要是增强脏数据的捕获能力,这里能够获取数据源以及数据源向下游算子报错时的脏数据。这样会减少手动剔除脏数据的情况 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
… and when the source sends it downstream #1901
Purpose of this pull request
Which issue you fix
Fixes # (issue).
Checklist: