同步操作将从 Apache SeaTunnel/SeaTunnel 强制同步,此操作会覆盖自 Fork 仓库以来所做的任何修改,且无法恢复!!!
确定后同步将在后台操作,完成时将刷新页面,请耐心等待。
Clickhouse file sink connector
Generate the clickhouse data file with the clickhouse-local program, and then send it to the clickhouse
server, also call bulk load. This connector only support clickhouse table which engine is 'Distributed'.And internal_replication
option
should be true
. Supports Batch and Streaming mode.
:::tip
Write data to Clickhouse can also be done using JDBC
:::
name | type | required | default value |
---|---|---|---|
host | string | yes | - |
database | string | yes | - |
table | string | yes | - |
username | string | yes | - |
password | string | yes | - |
clickhouse_local_path | string | yes | - |
sharding_key | string | no | - |
copy_method | string | no | scp |
node_free_password | boolean | no | false |
node_pass | list | no | - |
node_pass.node_address | string | no | - |
node_pass.username | string | no | "root" |
node_pass.password | string | no | - |
compatible_mode | boolean | no | false |
file_fields_delimiter | string | no | "\t" |
file_temp_path | string | no | "/tmp/seatunnel/clickhouse-local/file" |
common-options | no | - |
ClickHouse
cluster address, the format is host:port
, allowing multiple hosts
to be specified. Such as "host1:8123,host2:8123"
.
The ClickHouse
database
The table name
ClickHouse
user username
ClickHouse
user password
When ClickhouseFile split data, which node to send data to is a problem, the default is random selection, but the 'sharding_key' parameter can be used to specify the field for the sharding algorithm.
The address of the clickhouse-local program on the spark node. Since each task needs to be called, clickhouse-local should be located in the same path of each spark node.
Specifies the method used to transfer files, the default is scp, optional scp and rsync
Because seatunnel need to use scp or rsync for file transfer, seatunnel need clickhouse server-side access. If each spark node and clickhouse server are configured with password-free login, you can configure this option to true, otherwise you need to configure the corresponding node password in the node_pass configuration
Used to save the addresses and corresponding passwords of all clickhouse servers
The address corresponding to the clickhouse server
The username corresponding to the clickhouse server, default root user.
The password corresponding to the clickhouse server.
In the lower version of Clickhouse, the ClickhouseLocal program does not support the --path
parameter,
you need to use this mode to take other ways to realize the --path
parameter function
ClickhouseFile uses csv format to temporarily save data. If the data in the row contains the delimiter value of csv, it may cause program exceptions. Avoid this with this configuration. Value string has to be an exactly one character long
The directory where ClickhouseFile stores temporary files locally.
Sink plugin common parameters, please refer to Sink Common Options for details
ClickhouseFile {
host = "192.168.0.1:8123"
database = "default"
table = "fake_all"
username = "default"
password = ""
clickhouse_local_path = "/Users/seatunnel/Tool/clickhouse local"
sharding_key = "age"
node_free_password = false
node_pass = [{
node_address = "192.168.0.1"
password = "seatunnel"
}]
}
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。