diff options
| author | doufenghu <[email protected]> | 2024-01-27 12:21:28 +0800 |
|---|---|---|
| committer | doufenghu <[email protected]> | 2024-01-27 12:21:28 +0800 |
| commit | 38424d6655d952dcda4123f97aa66386312151e9 (patch) | |
| tree | 16605cc07f4fe5774c41a8867a1e106aebe4f44c /docs/connector | |
| parent | eb055c2917289b4ce8df0935a43b0b13d87bd561 (diff) | |
[Improve][docs] Improve 1.0.0 release docs.
Diffstat (limited to 'docs/connector')
| -rw-r--r-- | docs/connector/formats/protobuf.md | 19 | ||||
| -rw-r--r-- | docs/connector/sink/clickhouse.md (renamed from docs/connector/sink/ClickHouse.md) | 20 | ||||
| -rw-r--r-- | docs/connector/sink/kafka.md (renamed from docs/connector/sink/Kafka.md) | 0 | ||||
| -rw-r--r-- | docs/connector/sink/print.md (renamed from docs/connector/sink/Print.md) | 0 | ||||
| -rw-r--r-- | docs/connector/source/inline.md (renamed from docs/connector/source/Inline.md) | 0 | ||||
| -rw-r--r-- | docs/connector/source/ipfix.md (renamed from docs/connector/source/IPFIX.md) | 6 | ||||
| -rw-r--r-- | docs/connector/source/kafka.md (renamed from docs/connector/source/Kafka.md) | 0 |
7 files changed, 27 insertions, 18 deletions
diff --git a/docs/connector/formats/protobuf.md b/docs/connector/formats/protobuf.md index c798447..2efbeff 100644 --- a/docs/connector/formats/protobuf.md +++ b/docs/connector/formats/protobuf.md @@ -20,10 +20,19 @@ It is very popular in Streaming Data Pipeline. Now support protobuf format in so ## Data Type Mapping -| Data Type | Protobuf Data Type | Description | -|--------------|---------------------------------------------------------------------------------------------|-------------| -| int / bigint | int32 / int64 / uint32 / uint64 / sint32 / sint64 / fixed32 / fixed64 / sfixed32 / sfixed64 | | -| string | string | | +| Protobuf Data Type | Data Type | Description | +|-------------------------------------------------|-----------|------------------------------------------------------------------------------------------------------------------------| +| int32 / uint32 / sint32 / fixed32 / sfixed32 | int | Protobuf data type is recommended to use `int32 / sint32`. The data type also support `int / bigint / float / dobule`. | +| int64 / uint64 / sint64 / fixed64 / sfixed64 | bigint | Protobuf data type is recommended to use `int64 / sint64`. The data type also support `int / bigint / float / dobule`. | +| float | float | Protobuf data type is recommended to use `double`. The data type also support `int / bigint / float / dobule`. | +| double | double | Protobuf data type is recommended to use `double`. The data type also support `int / bigint / float / dobule`. | +| bool | boolean | Protobuf data type is recommended to use `int32`. The data type also support `boolean / int(0:false,1:true) `. | +| enum | int | Protobuf data type is recommended to use `int32`. The data type also support `int`. | +| string | string | In data serialization support all data type converted to `String`. | +| bytes | binary | - | +| message | struct | - | +| repeated | array | - | + # How to use ## protobuf uses example @@ -262,7 +271,7 @@ message SessionRecord { string tunnel_endpoint_b_desc = 223; } ``` -Build protobuf file to binary descriptor file. +Build protobuf file to binary descriptor file. Only support `proto3` syntax. If data type has null value, need add `optional` keyword. We recommend add `optional` keyword for int and double etc. ```shell protoc --descriptor_set_out=session_record_test.desc session_record_test.proto ``` diff --git a/docs/connector/sink/ClickHouse.md b/docs/connector/sink/clickhouse.md index 79ba1db..d794767 100644 --- a/docs/connector/sink/ClickHouse.md +++ b/docs/connector/sink/clickhouse.md @@ -32,16 +32,16 @@ In order to use the ClickHouse connector, the following dependencies are require ClickHouse sink custom properties. If properties belongs to ClickHouse JDBC Config, you can use `connection.` prefix to set. -| Name | Type | Required | Default | Description | -|-----------------------|----------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| host | String | Yes | - | `ClickHouse` cluster address, the format is `host:port` , allowing multiple `hosts` to be specified. Such as `"host1:8123,host2:8123"`. | -| database | String | Yes | - | The `ClickHouse` database. | -| table | String | Yes | - | The table name. | -| batch.size | Integer | Yes | 100000 | The number of rows written through [Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) each time, the `default is 20000`. | -| batch.interval | Duration | Yes | 30s | The time interval for writing data through. | -| connection.user | String | Yes | - | The username to use to connect to `ClickHouse`. | -| connection.password | String | Yes | - | The password to use to connect to `ClickHouse`. | -| connection.config | Map | No | - | In addition to the above mandatory parameters that must be specified by `clickhouse-jdbc` , users can also specify multiple optional parameters, which cover all the [parameters](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration) provided by `clickhouse-jdbc`. | +| Name | Type | Required | Default | Description | +|---------------------|----------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| host | String | Yes | - | `ClickHouse` cluster address, the format is `host:port` , allowing multiple `hosts` to be specified. Such as `"host1:8123,host2:8123"`. | +| connection.database | String | Yes | - | The `ClickHouse` database. | +| table | String | Yes | - | The table name. | +| batch.size | Integer | Yes | 100000 | The number of rows written through [Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) each time, the `default is 20000`. | +| batch.interval | Duration | Yes | 30s | The time interval for writing data through. | +| connection.user | String | Yes | - | The username to use to connect to `ClickHouse`. | +| connection.password | String | Yes | - | The password to use to connect to `ClickHouse`. | +| connection.config | Map | No | - | In addition to the above mandatory parameters that must be specified by `clickhouse-jdbc` , users can also specify multiple optional parameters, which cover all the [parameters](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration) provided by `clickhouse-jdbc`. | ## Example This example read data of inline test source and write to ClickHouse table `test`. diff --git a/docs/connector/sink/Kafka.md b/docs/connector/sink/kafka.md index 04dae2f..04dae2f 100644 --- a/docs/connector/sink/Kafka.md +++ b/docs/connector/sink/kafka.md diff --git a/docs/connector/sink/Print.md b/docs/connector/sink/print.md index 271d7a2..271d7a2 100644 --- a/docs/connector/sink/Print.md +++ b/docs/connector/sink/print.md diff --git a/docs/connector/source/Inline.md b/docs/connector/source/inline.md index c91d1a7..c91d1a7 100644 --- a/docs/connector/source/Inline.md +++ b/docs/connector/source/inline.md diff --git a/docs/connector/source/IPFIX.md b/docs/connector/source/ipfix.md index 550a5ab..424aa65 100644 --- a/docs/connector/source/IPFIX.md +++ b/docs/connector/source/ipfix.md @@ -13,9 +13,9 @@ IPFIX source custom properties. | Name | Type | Required | Default | Description | |-----------------------------------------|---------|----------|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| -| .port.range | Integer | Yes | - | UDP port range | -| buffer.size | Integer | No | 65535 | The maximum size of packet for UDP | -| receive.buffer.size | Integer | No | 104857600 | UDP receive buffer size in bytes | +| port.range | String | Yes | - | Range of ports such as 3000 -3010 | +| max.packet.size | Integer | No | 65535 | The maximum size of packet for UDP | +| max.receive.buffer.size | Integer | No | 104857600 | UDP maximum receive buffer size in bytes | | service.discovery.registry.mode | String | No | - | Service discovery registry mode, support `0(nacos)` and `1(consul)` | | service.discovery.service.name | String | No | - | Service discovery service name | | service.discovery.health.check.interval | Integer | No | - | Service discovery health check interval in milliseconds | diff --git a/docs/connector/source/Kafka.md b/docs/connector/source/kafka.md index 0565fd4..0565fd4 100644 --- a/docs/connector/source/Kafka.md +++ b/docs/connector/source/kafka.md |
