feat(connectors): add SurrealDB sink connector#3453
Open
countradooku wants to merge 3 commits into
Open
Conversation
SurrealDB is a document database target for Iggy connector users, so the sink writes batches with deterministic record ids and bulk INSERT IGNORE to keep runtime redelivery idempotent without per-message round trips. Constraint: User explicitly requested the latest SurrealDB Rust SDK and chose to keep it despite BUSL-1.1 license-validation warnings for SurrealDB crates. Constraint: Local Docker daemon was unavailable, so real-container integration execution could not run here. Rejected: Per-message SDK writes | too many round trips and weaker batching throughput. Rejected: Using the testcontainers SurrealDB module | module source hardcodes an older SurrealDB image. Confidence: medium Scope-risk: moderate Directive: Keep record ids deterministic across releases; changing build_record_id breaks replay idempotency. Tested: cargo fmt --all; cargo sort --no-format --workspace; cargo clippy --all-features --all-targets -- -D warnings; cargo check --all --all-features; cargo test -p iggy_connector_surrealdb_sink; cargo test -p integration --no-run connectors::surrealdb; cargo test --locked --doc; cargo doc --no-deps --all-features --quiet; taplo/license/shellcheck/version/diff/binary checks; prek install Not-tested: Docker-backed SurrealDB integration execution, because Docker daemon was not running locally.
7b8305a to
48c3a9d
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3453 +/- ##
============================================
- Coverage 74.57% 72.07% -2.50%
- Complexity 937 943 +6
============================================
Files 1249 1245 -4
Lines 123564 121326 -2238
Branches 99837 97627 -2210
============================================
- Hits 92143 87441 -4702
- Misses 28438 31098 +2660
+ Partials 2983 2787 -196
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a SurrealDB sink connector for writing Iggy messages into SurrealDB using the latest SurrealDB Rust SDK version available during implementation,
3.1.4.The connector supports deterministic record IDs, bulk
INSERT IGNOREwrites for idempotent replay, configurable batch sizing, root/namespace/database/no-auth modes, optional table and offset-index definition, payload modes (auto,json,text,base64), metadata/header/checksum/origin timestamp fields, retry/backoff handling, and runtime metrics logging.This also wires the connector into workspace membership, connector docs, example runtime config, binary artifact builds, edge-release output, version bump scripts, and Docker-backed integration test scaffolding.
Tests
cargo fmt --allcargo sort --no-format --workspacecargo clippy --all-features --all-targets -- -D warningscargo check --all --all-featurescargo test -p iggy_connector_surrealdb_sinkcargo test -p integration --no-run connectors::surrealdbcargo test --locked --doccargo doc --no-deps --all-features --quiet./scripts/ci/taplo.sh./scripts/ci/license-headers.sh./scripts/ci/shellcheck.sh./scripts/ci/binary-artifacts.sh --check./scripts/extract-version.sh --checkgit diff --checkprek installKnown Review Items
./scripts/ci/third-party-licenses.sh --validate --manifest core/connectors/sinks/surrealdb_sink/Cargo.tomlreports BUSL-1.1 license failures from SurrealDB SDK crates. This PR intentionally keeps the SDK dependency because the connector targets the latest SurrealDB SDK.