Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ Toto<sup><a href="#appendix5" id="ref5" style="text-decoration: none;">[5]</a></

![](/img/LargeModel-toto.png)

## 9. Performance Showcase
## 8. Performance Showcase

Time Series Large Models can adapt to real time series data from various different domains and scenarios, demonstrating excellent processing capabilities across various tasks. The following shows the actual performance on different datasets:

Expand All @@ -112,7 +112,7 @@ Using Time Series Large Models to accurately identify outliers that deviate sign

![](/img/LargeModel05.png)

## 10. Deployment and Usage
## 9. Deployment and Usage

1. Open the IoTDB CLI console and check that the ConfigNode, DataNode, and AINode nodes are all Running.

Expand Down
73 changes: 48 additions & 25 deletions src/UserGuide/Master/Table/User-Manual/Data-Sync_apache.md
Original file line number Diff line number Diff line change
Expand Up @@ -339,7 +339,7 @@ SQL Example:
CREATE PIPE A2B
WITH SINK (
'sink' = 'iotdb-thrift-sink',
'node-urls' = '127.0.0.1:6668' -- URL of the DataNode service port on the target IoTDB
'node-urls' = '127.0.0.1:6668'
)
```

Expand All @@ -357,17 +357,24 @@ SQL Example:
CREATE PIPE A2B
WITH SOURCE (
'source' = 'iotdb-source',
'mode.streaming' = 'true' -- Extraction mode for newly inserted data (after the pipe is created):
-- Whether to extract data in streaming mode (if set to false, batch mode is used).
'database-name'='testdb.*', -- Scope of Data Synchronization
'start-time' = '2023.08.23T08:00:00+00:00', -- The event time at which data synchronization starts (inclusive).
'end-time' = '2023.10.23T08:00:00+00:00' -- The event time at which data synchronization ends (inclusive).
'mode.streaming' = 'true',
'database-name'='testdb.*',
'start-time' = '2023.08.23T08:00:00+00:00',
'end-time' = '2023.10.23T08:00:00+00:00'
)
WITH SINK (
'sink' = 'iotdb-thrift-async-sink',
'node-urls' = '127.0.0.1:6668' -- The URL of the DataNode's data service port in the target IoTDB instance.
'node-urls' = '127.0.0.1:6668'
)
```

Where:

- `mode.streaming`: Extraction mode for newly inserted data (after the pipe is created): Whether to extract data in streaming mode (if set to false, batch mode is used).
- `database-name`: Scope of Data Synchronization
- `start-time`: The event time at which data synchronization starts (inclusive).
- `end-time`: The event time at which data synchronization ends (inclusive).

### 3.3 Edge-to-Cloud Data Transmission

This example demonstrates synchronizing data from multiple IoTDB clusters (B, C, D) to a central IoTDB cluster (A). The data pipeline is shown below:
Expand All @@ -381,12 +388,12 @@ SQL Example: On IoTDB B:
```SQL
CREATE PIPE BA
WITH SOURCE (
'database-name' = 'db_b.*', -- Restrict the database scope
'table-name' = '.*' -- Match all tables
'database-name' = 'db_b.*',
'table-name' = '.*'
)
WITH SINK (
'sink' = 'iotdb-thrift-sink',
'node-urls' = '127.0.0.1:6667' -- URL of the DataNode service port on the target IoTDB
'node-urls' = '127.0.0.1:6667'
)
```

Expand All @@ -395,12 +402,12 @@ On IoTDB C :
```SQL
CREATE PIPE CA
WITH SOURCE (
'database-name' = 'db_c.*', -- Restrict the database scope
'table-name' = '.*' -- Match all tables
'database-name' = 'db_c.*',
'table-name' = '.*'
)
WITH SINK (
'sink' = 'iotdb-thrift-sink',
'node-urls' = '127.0.0.1:6668' -- URL of the DataNode service port on the target IoTDB
'node-urls' = '127.0.0.1:6668'
)
```

Expand All @@ -409,29 +416,34 @@ On IoTDB D:
```SQL
CREATE PIPE DA
WITH SOURCE (
'database-name' = 'db_d.*', -- Restrict the database scope
'table-name' = '.*' -- Match all tables
'database-name' = 'db_d.*',
'table-name' = '.*'
)
WITH SINK (
'sink' = 'iotdb-thrift-sink',
'node-urls' = '127.0.0.1:6669' -- URL of the DataNode service port on the target IoTDB
'node-urls' = '127.0.0.1:6669'
)
```

The parameters in the preceding two examples have the same meanings as those below:

- `database-name`: Restrict the database scope
- `table-name`: Match all tables
- `node-urls`: URL of the DataNode service port on the target IoTDB

### 3.4 Cascaded Data Transmission

This example demonstrates cascading data transmission from IoTDB A to IoTDB B and then to IoTDB C. The data pipeline is shown below:

![](/img/sync_en_04.png)


SQL Example: On IoTDB A:

```SQL
CREATE PIPE AB
WITH SINK (
'sink' = 'iotdb-thrift-sink',
'node-urls' = '127.0.0.1:6668' -- URL of the DataNode service port on the target IoTDB
'node-urls' = '127.0.0.1:6668'
)
```

Expand All @@ -443,10 +455,13 @@ WITH SOURCE (
)
WITH SINK (
'sink' = 'iotdb-thrift-sink',
'node-urls' = '127.0.0.1:6669' -- URL of the DataNode service port on the target IoTDB
'node-urls' = '127.0.0.1:6669'
)
```

The parameters in the preceding example have the same meanings as those below:

- `node-urls`: URL of the DataNode service port on the target IoTDB

### 3.5 Compressed Synchronization

Expand All @@ -457,12 +472,15 @@ IoTDB supports specifying data compression methods during synchronization. The `
```SQL
CREATE PIPE A2B
WITH SINK (
'node-urls' = '127.0.0.1:6668', -- URL of the DataNode service port on the target IoTDB
'compressor' = 'snappy,lz4', -- Compression algorithms
'rate-limit-bytes-per-second' = '1048576' -- Maximum bytes allowed per second
'node-urls' = '127.0.0.1:6668',
'compressor' = 'snappy,lz4',
'rate-limit-bytes-per-second' = '1048576'
)
```

Where:

- `node-urls`: URL of the DataNode service port on the target IoTDB

### 3.6 Encrypted Synchronization

Expand All @@ -474,12 +492,17 @@ IoTDB supports SSL encryption during synchronization to securely transmit data b
CREATE PIPE A2B
WITH SINK (
'sink' = 'iotdb-thrift-ssl-sink',
'node-urls' = '127.0.0.1:6667', -- URL of the DataNode service port on the target IoTDB
'ssl.trust-store-path' = 'pki/trusted', -- Path to the trust store certificate
'ssl.trust-store-pwd' = 'root' -- Password for the trust store certificate
'node-urls' = '127.0.0.1:6667',
'ssl.trust-store-path' = 'pki/trusted',
'ssl.trust-store-pwd' = 'root'
)
```

Where:

- `node-urls`: URL of the DataNode service port on the target IoTDB
- `ssl.trust-store-path`: The trust store certificate path required to connect to the target DataNode
- `ssl.trust-store-pwd`: The trust store certificate password required to connect to the target DataNode

## Reference: Notes

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ Toto<sup><a href="#appendix5" id="ref5" style="text-decoration: none;">[5]</a></

![](/img/LargeModel-toto.png)

## 9. Performance Showcase
## 8. Performance Showcase

Time Series Large Models can adapt to real time series data from various different domains and scenarios, demonstrating excellent processing capabilities across various tasks. The following shows the actual performance on different datasets:

Expand All @@ -112,7 +112,7 @@ Using Time Series Large Models to accurately identify outliers that deviate sign

![](/img/LargeModel05.png)

## 10. Deployment and Usage
## 9. Deployment and Usage

1. Open the IoTDB CLI console and check that the ConfigNode, DataNode, and AINode nodes are all Running.

Expand Down
73 changes: 50 additions & 23 deletions src/UserGuide/Master/Tree/User-Manual/Data-Sync_apache.md
Original file line number Diff line number Diff line change
Expand Up @@ -365,7 +365,7 @@ In this example, we can create a synchronization task named A2B to synchronize t
create pipe A2B
with sink (
'sink'='iotdb-thrift-sink',
'node-urls' = '127.0.0.1:6668', -- The URL of the data service port of the DataNode node on the target IoTDB
'node-urls' = '127.0.0.1:6668',
```

### 3.2 Partial data synchronization
Expand All @@ -382,17 +382,24 @@ The detailed statements are as follows:
create pipe A2B
WITH SOURCE (
'source'= 'iotdb-source',
'realtime.mode' = 'stream' -- The extraction mode for newly inserted data (after pipe creation)
'path' = 'root.vehicle.**', -- Scope of Data Synchronization
'start-time' = '2023.08.23T08:00:00+00:00', -- The start event time for synchronizing all data, including start-time
'end-time' = '2023.10.23T08:00:00+00:00' -- The end event time for synchronizing all data, including end-time
'realtime.mode' = 'stream',
'path' = 'root.vehicle.**',
'start-time' = '2023.08.23T08:00:00+00:00',
'end-time' = '2023.10.23T08:00:00+00:00'
)
with SINK (
'sink'='iotdb-thrift-async-sink',
'node-urls' = '127.0.0.1:6668', -- The URL of the data service port of the DataNode node on the target IoTDB
'node-urls' = '127.0.0.1:6668',
)
```

Where:

- `realtime.mode`: The extraction mode for newly inserted data (after pipe creation)
- `path`: Scope of Data Synchronization
- `start-time`: The start event time for synchronizing all data, including start-time
- `end-time`: The end event time for synchronizing all data, including end-time

### 3.3 Edge-cloud data transfer

This example is used to demonstrate the scenario where data from multiple IoTDB is transferred to the cloud, with data from clusters B, C, and D all synchronized to cluster A, as shown in the figure below:
Expand All @@ -406,12 +413,12 @@ On B IoTDB, execute the following statement to synchronize data from B to A:
```SQL
create pipe BA
with source (
'inclusion'='all', -- Indicates synchronization of full data, schema , and auth
'path'='root.db.**', -- Limit the range
'inclusion'='all',
'path'='root.db.**',
)
with sink (
'sink'='iotdb-thrift-sink',
'node-urls' = '127.0.0.1:6668', -- The URL of the data service port of the DataNode node on the target IoTDB
'node-urls' = '127.0.0.1:6668',
)
)
```
Expand All @@ -421,11 +428,11 @@ On C IoTDB, execute the following statement to synchronize data from C to A:
```SQL
create pipe CA
with source (
'inclusion'='all', -- Indicates synchronization of full data, schema , and auth
'path'='root.db.**', -- Limit the range
'inclusion'='all',
'path'='root.db.**',
with sink (
'sink'='iotdb-thrift-sink',
'node-urls' = '127.0.0.1:6668', -- The URL of the data service port of the DataNode node on the target IoTDB
'node-urls' = '127.0.0.1:6668',
)
)
```
Expand All @@ -435,16 +442,22 @@ On D IoTDB, execute the following statement to synchronize data from D to A:
```SQL
create pipe DA
with source (
'inclusion'='all', -- Indicates synchronization of full data, schema , and auth
'path'='root.db.**', -- Limit the range
'inclusion'='all',
'path'='root.db.**',
)
with sink (
'sink'='iotdb-thrift-sink',
'node-urls' = '127.0.0.1:6668', -- The URL of the data service port of the DataNode node on the target IoTDB
'node-urls' = '127.0.0.1:6668',
)
)
```

The parameters in the preceding two examples have the same meanings as those below:

- `inclusion`: Indicates synchronization of full data, schema , and auth
- `path`: Limit the range
- `node-urls`: The URL of the data service port of the DataNode node on the target IoTDB

### 3.4 Cascading data transfer

This example is used to demonstrate the scenario where data is transferred in a cascading manner between multiple IoTDB, with data from cluster A synchronized to cluster B, and then to cluster C, as shown in the figure below:
Expand All @@ -459,7 +472,7 @@ On A IoTDB, execute the following statement to synchronize data from A to B:
create pipe AB
with sink (
'sink'='iotdb-thrift-sink',
'node-urls' = '127.0.0.1:6668', -- The URL of the data service port of the DataNode node on the target IoTDB
'node-urls' = '127.0.0.1:6668',
)
)
```
Expand All @@ -469,15 +482,19 @@ On B IoTDB, execute the following statement to synchronize data from B to C:
```SQL
create pipe BC
with source (
'forwarding-pipe-requests' = 'true' -- Whether to forward data written by other Pipes
'forwarding-pipe-requests' = 'true'
)
with sink (
'sink'='iotdb-thrift-sink',
'node-urls' = '127.0.0.1:6669', -- The URL of the data service port of the DataNode node on the target IoTDB
'node-urls' = '127.0.0.1:6669',
)
)
```

The `node-urls` parameter in the preceding example has the same meaning as below:

- `forwarding-pipe-requests`: Whether to forward data written by other Pipes
- `node-urls`: The URL of the data service port of the DataNode node on the target IoTDB

### 3.5 Compression Synchronization (V1.3.3+)

Expand All @@ -488,11 +505,15 @@ For example, to create a synchronization task named A2B:
```SQL
create pipe A2B
with sink (
'node-urls' = '127.0.0.1:6668', -- The URL of the data service port of the DataNode node on the target IoTDB
'compressor' = 'snappy,lz4' -- Compression algorithms
'node-urls' = '127.0.0.1:6668',
'compressor' = 'snappy,lz4'
)
```

Where:

- `node-urls`: The URL of the data service port of the DataNode node on the target IoTDB

### 3.6 Encrypted Synchronization (V1.3.1+)

IoTDB supports the use of SSL encryption during the synchronization process, ensuring the secure transfer of data between different IoTDB instances. By configuring SSL-related parameters, such as the certificate address and password (`ssl.trust-store-path`)、(`ssl.trust-store-pwd`), data can be protected by SSL encryption during the synchronization process.
Expand All @@ -503,12 +524,18 @@ For example, to create a synchronization task named A2B:
create pipe A2B
with sink (
'sink'='iotdb-thrift-ssl-sink',
'node-urls'='127.0.0.1:6667', -- The URL of the data service port of the DataNode node on the target IoTDB
'ssl.trust-store-path'='pki/trusted', -- The trust store certificate path required to connect to the target DataNode
'ssl.trust-store-pwd'='root' -- The trust store certificate password required to connect to the target DataNode
'node-urls'='127.0.0.1:6667',
'ssl.trust-store-path'='pki/trusted',
'ssl.trust-store-pwd'='root'
)
```

Where:

- `node-urls`: The URL of the data service port of the DataNode node on the target IoTDB
- `ssl.trust-store-path`: The trust store certificate path required to connect to the target DataNode
- `ssl.trust-store-pwd`: The trust store certificate password required to connect to the target DataNode

## 4. Reference: Notes

You can adjust the parameters for data synchronization by modifying the IoTDB configuration file (`iotdb-system.properties`), such as the directory for storing synchronized data. The complete configuration is as follows:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ Toto<sup><a href="#appendix5" id="ref5" style="text-decoration: none;">[5]</a></

![](/img/LargeModel-toto.png)

## 9. Performance Showcase
## 8. Performance Showcase

Time Series Large Models can adapt to real time series data from various different domains and scenarios, demonstrating excellent processing capabilities across various tasks. The following shows the actual performance on different datasets:

Expand All @@ -112,7 +112,7 @@ Using Time Series Large Models to accurately identify outliers that deviate sign

![](/img/LargeModel05.png)

## 10. Deployment and Usage
## 9. Deployment and Usage

1. Open the IoTDB CLI console and check that the ConfigNode, DataNode, and AINode nodes are all Running.

Expand Down
Loading