summaryrefslogtreecommitdiff
path: root/docs/connector
diff options
context:
space:
mode:
authordoufenghu <[email protected]>2024-01-27 12:21:28 +0800
committerdoufenghu <[email protected]>2024-01-27 12:21:28 +0800
commit38424d6655d952dcda4123f97aa66386312151e9 (patch)
tree16605cc07f4fe5774c41a8867a1e106aebe4f44c /docs/connector
parenteb055c2917289b4ce8df0935a43b0b13d87bd561 (diff)
[Improve][docs] Improve 1.0.0 release docs.
Diffstat (limited to 'docs/connector')
-rw-r--r--docs/connector/formats/protobuf.md19
-rw-r--r--docs/connector/sink/clickhouse.md (renamed from docs/connector/sink/ClickHouse.md)20
-rw-r--r--docs/connector/sink/kafka.md (renamed from docs/connector/sink/Kafka.md)0
-rw-r--r--docs/connector/sink/print.md (renamed from docs/connector/sink/Print.md)0
-rw-r--r--docs/connector/source/inline.md (renamed from docs/connector/source/Inline.md)0
-rw-r--r--docs/connector/source/ipfix.md (renamed from docs/connector/source/IPFIX.md)6
-rw-r--r--docs/connector/source/kafka.md (renamed from docs/connector/source/Kafka.md)0
7 files changed, 27 insertions, 18 deletions
diff --git a/docs/connector/formats/protobuf.md b/docs/connector/formats/protobuf.md
index c798447..2efbeff 100644
--- a/docs/connector/formats/protobuf.md
+++ b/docs/connector/formats/protobuf.md
@@ -20,10 +20,19 @@ It is very popular in Streaming Data Pipeline. Now support protobuf format in so
## Data Type Mapping
-| Data Type | Protobuf Data Type | Description |
-|--------------|---------------------------------------------------------------------------------------------|-------------|
-| int / bigint | int32 / int64 / uint32 / uint64 / sint32 / sint64 / fixed32 / fixed64 / sfixed32 / sfixed64 | |
-| string | string | |
+| Protobuf Data Type | Data Type | Description |
+|-------------------------------------------------|-----------|------------------------------------------------------------------------------------------------------------------------|
+| int32 / uint32 / sint32 / fixed32 / sfixed32 | int | Protobuf data type is recommended to use `int32 / sint32`. The data type also support `int / bigint / float / dobule`. |
+| int64 / uint64 / sint64 / fixed64 / sfixed64 | bigint | Protobuf data type is recommended to use `int64 / sint64`. The data type also support `int / bigint / float / dobule`. |
+| float | float | Protobuf data type is recommended to use `double`. The data type also support `int / bigint / float / dobule`. |
+| double | double | Protobuf data type is recommended to use `double`. The data type also support `int / bigint / float / dobule`. |
+| bool | boolean | Protobuf data type is recommended to use `int32`. The data type also support `boolean / int(0:false,1:true) `. |
+| enum | int | Protobuf data type is recommended to use `int32`. The data type also support `int`. |
+| string | string | In data serialization support all data type converted to `String`. |
+| bytes | binary | - |
+| message | struct | - |
+| repeated | array | - |
+
# How to use
## protobuf uses example
@@ -262,7 +271,7 @@ message SessionRecord {
string tunnel_endpoint_b_desc = 223;
}
```
-Build protobuf file to binary descriptor file.
+Build protobuf file to binary descriptor file. Only support `proto3` syntax. If data type has null value, need add `optional` keyword. We recommend add `optional` keyword for int and double etc.
```shell
protoc --descriptor_set_out=session_record_test.desc session_record_test.proto
```
diff --git a/docs/connector/sink/ClickHouse.md b/docs/connector/sink/clickhouse.md
index 79ba1db..d794767 100644
--- a/docs/connector/sink/ClickHouse.md
+++ b/docs/connector/sink/clickhouse.md
@@ -32,16 +32,16 @@ In order to use the ClickHouse connector, the following dependencies are require
ClickHouse sink custom properties. If properties belongs to ClickHouse JDBC Config, you can use `connection.` prefix to set.
-| Name | Type | Required | Default | Description |
-|-----------------------|----------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| host | String | Yes | - | `ClickHouse` cluster address, the format is `host:port` , allowing multiple `hosts` to be specified. Such as `"host1:8123,host2:8123"`. |
-| database | String | Yes | - | The `ClickHouse` database. |
-| table | String | Yes | - | The table name. |
-| batch.size | Integer | Yes | 100000 | The number of rows written through [Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) each time, the `default is 20000`. |
-| batch.interval | Duration | Yes | 30s | The time interval for writing data through. |
-| connection.user | String | Yes | - | The username to use to connect to `ClickHouse`. |
-| connection.password | String | Yes | - | The password to use to connect to `ClickHouse`. |
-| connection.config | Map | No | - | In addition to the above mandatory parameters that must be specified by `clickhouse-jdbc` , users can also specify multiple optional parameters, which cover all the [parameters](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration) provided by `clickhouse-jdbc`. |
+| Name | Type | Required | Default | Description |
+|---------------------|----------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| host | String | Yes | - | `ClickHouse` cluster address, the format is `host:port` , allowing multiple `hosts` to be specified. Such as `"host1:8123,host2:8123"`. |
+| connection.database | String | Yes | - | The `ClickHouse` database. |
+| table | String | Yes | - | The table name. |
+| batch.size | Integer | Yes | 100000 | The number of rows written through [Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) each time, the `default is 20000`. |
+| batch.interval | Duration | Yes | 30s | The time interval for writing data through. |
+| connection.user | String | Yes | - | The username to use to connect to `ClickHouse`. |
+| connection.password | String | Yes | - | The password to use to connect to `ClickHouse`. |
+| connection.config | Map | No | - | In addition to the above mandatory parameters that must be specified by `clickhouse-jdbc` , users can also specify multiple optional parameters, which cover all the [parameters](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration) provided by `clickhouse-jdbc`. |
## Example
This example read data of inline test source and write to ClickHouse table `test`.
diff --git a/docs/connector/sink/Kafka.md b/docs/connector/sink/kafka.md
index 04dae2f..04dae2f 100644
--- a/docs/connector/sink/Kafka.md
+++ b/docs/connector/sink/kafka.md
diff --git a/docs/connector/sink/Print.md b/docs/connector/sink/print.md
index 271d7a2..271d7a2 100644
--- a/docs/connector/sink/Print.md
+++ b/docs/connector/sink/print.md
diff --git a/docs/connector/source/Inline.md b/docs/connector/source/inline.md
index c91d1a7..c91d1a7 100644
--- a/docs/connector/source/Inline.md
+++ b/docs/connector/source/inline.md
diff --git a/docs/connector/source/IPFIX.md b/docs/connector/source/ipfix.md
index 550a5ab..424aa65 100644
--- a/docs/connector/source/IPFIX.md
+++ b/docs/connector/source/ipfix.md
@@ -13,9 +13,9 @@ IPFIX source custom properties.
| Name | Type | Required | Default | Description |
|-----------------------------------------|---------|----------|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| .port.range | Integer | Yes | - | UDP port range |
-| buffer.size | Integer | No | 65535 | The maximum size of packet for UDP |
-| receive.buffer.size | Integer | No | 104857600 | UDP receive buffer size in bytes |
+| port.range | String | Yes | - | Range of ports such as 3000 -3010 |
+| max.packet.size | Integer | No | 65535 | The maximum size of packet for UDP |
+| max.receive.buffer.size | Integer | No | 104857600 | UDP maximum receive buffer size in bytes |
| service.discovery.registry.mode | String | No | - | Service discovery registry mode, support `0(nacos)` and `1(consul)` |
| service.discovery.service.name | String | No | - | Service discovery service name |
| service.discovery.health.check.interval | Integer | No | - | Service discovery health check interval in milliseconds |
diff --git a/docs/connector/source/Kafka.md b/docs/connector/source/kafka.md
index 0565fd4..0565fd4 100644
--- a/docs/connector/source/Kafka.md
+++ b/docs/connector/source/kafka.md