Skip to content

feat(connectors): Clickhouse Sink Connector#2886

Open
kriti-sc wants to merge 56 commits into
apache:masterfrom
kriti-sc:clickhouse-sink
Open

feat(connectors): Clickhouse Sink Connector#2886
kriti-sc wants to merge 56 commits into
apache:masterfrom
kriti-sc:clickhouse-sink

Conversation

@kriti-sc

@kriti-sc kriti-sc commented Mar 6, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2539

Rationale

Clickhouse is a real-time data analytics engine, and very popular in modern analytics architectures.

What changed?

This PR introduces a Clickhouse Sink Connector that enables writing data from Iggy to Clickhouse.

The Clickhouse writing logic is heavily inspired by the official Clickhouse Kafka Connector.

Local Execution

  • Produced messages 30456 + 29060 rows with schema user_id: String, user_type: u8, email: String, source: String, state: String, created_at: DateTime, message: String using sample data producer.
  • Consumed messages using the Clickhouse sink and into the particular Clickhouse table.
  • Verified schema and number of rows in Clickhouse.
  • Added unit tests and e2e tests, both passing.

Images 1&2: Produced 30456 + 29060 rows into Iggy in two batches
Image 3: Verified schema and number of rows in Clickhouse

image image image

AI Usage

  1. Which tools? (e.g., GitHub Copilot, Claude, ChatGPT) Claude Code
  2. Scope of usage? (e.g., autocomplete, generated functions, entire implementation) generated functions
  3. How did you verify the generated code works correctly? Manual testing by producing data into Iggy and then running the sink and verifying insertion into Clickhouse, unit tests and e2e tests for different Clickhouse insert configurations.
  4. Can you explain every line of the code if asked? Yes

@codecov

codecov Bot commented Mar 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.56210% with 358 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.10%. Comparing base (d254604) to head (bc7ce0a).
⚠️ Report is 5 commits behind head on master.

Files with missing lines Patch % Lines
...ore/connectors/sinks/clickhouse_sink/src/binary.rs 82.95% 122 Missing and 56 partials ⚠️
...ore/connectors/sinks/clickhouse_sink/src/schema.rs 76.45% 30 Missing and 75 partials ⚠️
...ore/connectors/sinks/clickhouse_sink/src/client.rs 72.68% 55 Missing and 10 partials ⚠️
core/connectors/sinks/clickhouse_sink/src/lib.rs 97.27% 3 Missing and 1 partial ⚠️
core/connectors/sinks/clickhouse_sink/src/sink.rs 60.00% 4 Missing ⚠️
core/connectors/sinks/clickhouse_sink/src/body.rs 98.80% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #2886       +/-   ##
=============================================
- Coverage     74.43%   48.10%   -26.34%     
  Complexity      943      943               
=============================================
  Files          1245     1249        +4     
  Lines        121293   108413    -12880     
  Branches      97599    84748    -12851     
=============================================
- Hits          90285    52149    -38136     
- Misses        28054    53445    +25391     
+ Partials       2954     2819      -135     
Components Coverage Δ
Rust Core 41.69% <82.56%> (-33.88%) ⬇️
Java SDK 58.44% <ø> (ø)
C# SDK 69.44% <ø> (-0.48%) ⬇️
Python SDK 81.06% <ø> (ø)
PHP SDK 83.57% <ø> (ø)
Node SDK 91.22% <ø> (-0.13%) ⬇️
Go SDK 40.20% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/sinks/clickhouse_sink/src/body.rs 98.80% <98.80%> (ø)
core/connectors/sinks/clickhouse_sink/src/lib.rs 97.27% <97.27%> (ø)
core/connectors/sinks/clickhouse_sink/src/sink.rs 60.00% <60.00%> (ø)
...ore/connectors/sinks/clickhouse_sink/src/client.rs 72.68% <72.68%> (ø)
...ore/connectors/sinks/clickhouse_sink/src/schema.rs 76.45% <76.45%> (ø)
...ore/connectors/sinks/clickhouse_sink/src/binary.rs 82.95% <82.95%> (ø)

... and 350 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs
@kriti-sc kriti-sc requested a review from abonander March 7, 2026 10:58
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

@github-actions github-actions Bot added S-stale Inactive issue or pull request and removed S-stale Inactive issue or pull request labels Mar 17, 2026

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall good direction, just needs a little bit polishing

Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/schema.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/client.rs
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated
@github-actions

github-actions Bot commented Apr 2, 2026

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

@github-actions github-actions Bot added S-stale Inactive issue or pull request and removed S-stale Inactive issue or pull request labels Apr 2, 2026
@kriti-sc kriti-sc requested a review from hubcio April 3, 2026 19:56
@github-actions

Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

@github-actions github-actions Bot added S-stale Inactive issue or pull request and removed S-stale Inactive issue or pull request labels Apr 18, 2026
@kriti-sc

kriti-sc commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement ClickHouse Sink Connector

5 participants