feat: implement REST API database and table CRUD operations with DLF authentication by discivigour · Pull Request #147 · apache/paimon-rust

discivigour · 2026-03-20T06:59:42Z

Purpose

Linked issue: #119

This PR implements complete REST API CRUD operations and DLF authentication for the Paimon Rust project.

Key Features:

Complete REST API database operations (list_databases, create_database, get_database, alter_database, drop_database)
Complete REST API table operations (list_tables, create_table, get_table, rename_table, drop_table)
DLF (Alibaba Cloud Data Lake Formation) authentication provider implementation

Brief change log

Core Feature Implementation

REST API CRUD Operations
- Implement complete database CRUD operations: list_databases, create_database, get_database, alter_database, drop_database
- Implement complete table CRUD operations: list_tables, create_table, get_table, rename_table, drop_table
- Unified API request builder and response handling
DLF Authentication Support
- Implement Alibaba Cloud DLF (Data Lake Formation) authentication provider
- Complete Alibaba Cloud signature algorithm implementation
- Support Token authentication method
- Configurable authentication provider factory

Tests

# Run REST API related tests
cargo test --package paimon --test rest_api_test

API and Format

Documentation

…tApi-crud

…tion - Add full REST API support for database and table operations - Implement DLF (Alibaba Cloud Data Lake Formation) authentication provider - Add bearer token authentication support - Refactor mock server for better testing - Add comprehensive test coverage for REST API operations - Fix clippy warnings and apply code formatting Database operations: - list_databases, create_database, get_database, alter_database, drop_database Table operations: - list_tables, create_table, get_table, rename_table, drop_table Authentication: - DLF authentication with signature generation - Bearer token authentication - Configurable token provider factory

- Rename BearTokenAuthProvider to BearerTokenAuthProvider for correctness - Update all references in auth module and API exports - Fix typo in naming to match Bearer token authentication standard

luoyuxia

@discivigour Haven't review all files. But left some comments firstly.

luoyuxia · 2026-03-23T09:31:14Z

Makefile

 	docker compose -f dev/docker-compose.yaml down -v
+
+# Code quality checks
+check:


why add in this pr? I'd perfer to make the pr focus

luoyuxia · 2026-03-23T09:31:23Z

.licenserc.yaml

+    spdx-id: Apache-2.0
+    copyright-owner: Apache Software Foundation
+
+  paths-ignore:


why add in this pr? I'd perfer to make the pr focus

luoyuxia · 2026-03-23T09:32:02Z

.gitignore

 .vscode
 **/.DS_Store
 dist/*
+crates/paimon/examples/dlf_*.rs


I think we can remove this since seem we have no dlf_*.rs

luoyuxia · 2026-03-23T09:32:18Z

dev/check.sh

+# specific language governing permissions and limitations
+# under the License.
+
+set -e


why add in this pr? I'd perfer to make the pr focus

ok, I will add the shell script in a separate pr.

luoyuxia

@discivigour Thanks for the pr. Left some comments. PTAL

luoyuxia · 2026-03-23T12:21:04Z

crates/paimon/examples/rest_crud.rs

+
+    // List databases
+    println!("Listing databases...");
+    match api.list_databases().await {


I'm concerned about introduing this example. Do use really care about using api ? Users only need to care about the catalog interface. So, two suggestion in here

remove this example, and add example in following pr if we found it's useful

keep this example, but use catalog interface.

Got it, I will remove it and add examples of catalog interface in next pr.

luoyuxia · 2026-03-23T12:24:16Z

crates/paimon/src/api/auth/dlf_provider.rs

+        access_key_id: impl Into<String>,
+        access_key_secret: impl Into<String>,
+        security_token: Option<String>,
+        expiration: Option<String>,


nit:
change the order to

expiration_at_millis: Option<i64>, expiration: Option<String>,

since it prefer expiration_at_millis

luoyuxia · 2026-03-23T12:29:36Z

crates/paimon/src/api/auth/dlf_provider.rs

+                .get(CatalogOptions::DLF_TOKEN_ECS_METADATA_URL)
+                .cloned()
+                .unwrap_or_else(|| {
+                    "http://100.100.100.200/latest/meta-data/Ram/security-credentials/".to_string()


just double check what's http://100.100.100.200. Is it correct to hardcode it as the default value?

I have checked it. It's defaut value for aliyun ecs token auth.

luoyuxia · 2026-03-23T12:37:15Z

crates/paimon/src/api/auth/dlf_provider.rs

+    /// - Load a new token if current token is None
+    /// - Refresh the token if it's about to expire (within TOKEN_EXPIRATION_SAFE_TIME_MILLIS)
+    fn get_token(&self) -> Result<DLFToken, String> {
+        if let Some(ref loader) = self.token_loader {


nit:

if let Some(loader) = &self.token_loader { let need_reload = { let token = self.token.borrow(); match token.as_ref() { None => true, Some(token) => match token.expiration_at_millis { Some(expiration_at_millis) => { let now = chrono::Utc::now().timestamp_millis(); expiration_at_millis - now < TOKEN_EXPIRATION_SAFE_TIME_MILLIS } None => false, }, } }; if need_reload { let new_token = loader.load_token()?; *self.token.borrow_mut() = Some(new_token); } }

Change this to make it more idiomatic Rust.

luoyuxia · 2026-03-23T12:38:39Z

crates/paimon/src/api/auth/dlf_provider.rs

+    /// If token_loader is configured, this method will:
+    /// - Load a new token if current token is None
+    /// - Refresh the token if it's about to expire (within TOKEN_EXPIRATION_SAFE_TIME_MILLIS)
+    fn get_token(&self) -> Result<DLFToken, String> {


rename to. get_or_refresh_token?
Also, I'm wondering can it be mut self so that we don't need RefCell<Option<DLFToken>>, RefCell feel a little hack to me considering it's not friendly to async block

Agreed. I will change it.

luoyuxia · 2026-03-23T13:18:04Z

crates/paimon/src/api/auth/dlf_provider.rs

+        let token = match self.get_token() {
+            Ok(t) => t,
+            Err(e) => {
+                eprintln!("Failed to get token: {}", e);


Is it expected we can just ignore this error and return base_header diectly?
If so, use log::info instead of eprintln

luoyuxia · 2026-03-23T13:24:36Z

crates/paimon/src/api/auth/dlf_provider.rs

+    }
+
+    /// Perform HTTP GET request with retry logic (sync version).
+    fn get(&self, url: &str) -> Result<String, String> {


we always use the Result define in paimon, please use crates/paimon/src/error.rs

luoyuxia · 2026-03-23T13:26:34Z

crates/paimon/src/api/api_request.rs

+use crate::{catalog::Identifier, spec::Schema};
+
+/// Base trait for REST requests.
+pub trait RESTRequest {}


do we really need this trait currently?

luoyuxia · 2026-03-23T13:31:33Z

crates/paimon/src/api/auth/dlf_provider.rs

+        token_loader: Option<Arc<dyn DLFTokenLoader>>,
+    ) -> Self {
+        if token.is_none() && token_loader.is_none() {
+            panic!("Either token or token_loader must be provided");


not panic in here, return error instead

luoyuxia · 2026-03-23T13:32:38Z

crates/paimon/src/api/resource_paths.rs

+            "{}/{}/{}/{}",
+            self.base_path,
+            Self::DATABASES,
+            database_name,


shoudn't it do encode_string?

You're right, I missed that.

XiaoHongbo-Hope · 2026-03-24T11:56:03Z

crates/paimon/src/api/auth/factory.rs

+            let host = host.split('/').next().unwrap_or("");
+            let host = host.split(':').next().unwrap_or("");
+
+            if host.starts_with("dlfnext") || host.contains("openapi") {


host.contains("openapi") should not be needed, I will check with Tu Ming.

XiaoHongbo-Hope · 2026-03-24T12:00:23Z

crates/paimon/src/api/auth/dlf_provider.rs

+            Ok(t) => t,
+            Err(e) => {
+                eprintln!("Failed to get token: {}", e);
+                return base_header;


I suggest to keep same behavior with java, throwing exception when failure.

Agreed. I will change it.

…tApi-crud

umi added 4 commits March 18, 2026 19:28

proto

1b36dc5

Merge branch 'main' of https://github.com/apache/paimon-rust into res…

d01cfcc

…tApi-crud

fix

3e7fba8

discivigour marked this pull request as draft March 20, 2026 07:04

discivigour changed the title ~~feat: implement complete REST API CRUD operations with DLF authentication~~ feat: implement REST API database and table CRUD operations with DLF authentication Mar 20, 2026

umi added 14 commits March 20, 2026 15:19

fix

103ebd6

fix

6713d5b

fix

5ee98a5

rm

0459be2

refactor: rename BearTokenAuthProvider to BearerTokenAuthProvider

2745c52

- Rename BearTokenAuthProvider to BearerTokenAuthProvider for correctness - Update all references in auth module and API exports - Fix typo in naming to match Bearer token authentication standard

dynamicToken

7c3cee3

tokenRefresh

fa85f28

fmt

b903331

addShell

bd095d8

fix

f8a3731

fix

53def9e

fix

5fe64cd

fix

6660624

fix

e00493a

discivigour marked this pull request as ready for review March 23, 2026 07:21

luoyuxia reviewed Mar 23, 2026

View reviewed changes

umi added 2 commits March 23, 2026 18:24

chore: remove check script changes (moved to add-check-script branch)

66a4a61

rm

0c9f7d9

luoyuxia reviewed Mar 23, 2026

View reviewed changes

XiaoHongbo-Hope reviewed Mar 24, 2026

View reviewed changes

umi added 3 commits March 25, 2026 14:32

fix

c7b9427

Merge branch 'main' of https://github.com/apache/paimon-rust into res…

e34df9b

…tApi-crud

rm

6d0a982

Conversation

discivigour commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Brief change log

Core Feature Implementation

Tests

API and Format

Documentation

Uh oh!

luoyuxia left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luoyuxia left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

discivigour commented Mar 20, 2026 •

edited

Loading

XiaoHongbo-Hope Mar 24, 2026 •

edited

Loading