summaryrefslogtreecommitdiff
path: root/docs/connector
diff options
context:
space:
mode:
authordoufenghu <[email protected]>2024-05-02 17:14:05 +0800
committerdoufenghu <[email protected]>2024-05-02 17:14:05 +0800
commita172d68a345cd1943001533be9209fe04a9d5581 (patch)
tree13ad16eb846f5694ef7f37a5888243361d99f37d /docs/connector
parent27eb76f576eb0ab4d611cab72cb3fca01eac5cec (diff)
[Improve][docs] Add support feature documents from 1.2.1 to 1.3.1
Diffstat (limited to 'docs/connector')
-rw-r--r--docs/connector/connector.md20
-rw-r--r--docs/connector/formats/json.md8
-rw-r--r--docs/connector/formats/msgpack.md12
-rw-r--r--docs/connector/formats/protobuf.md6
-rw-r--r--docs/connector/formats/raw.md2
-rw-r--r--docs/connector/sink/clickhouse.md18
-rw-r--r--docs/connector/sink/kafka.md25
-rw-r--r--docs/connector/sink/print.md4
-rw-r--r--docs/connector/source/inline.md6
-rw-r--r--docs/connector/source/ipfix.md24
-rw-r--r--docs/connector/source/kafka.md15
11 files changed, 78 insertions, 62 deletions
diff --git a/docs/connector/connector.md b/docs/connector/connector.md
index e36214c..7031bed 100644
--- a/docs/connector/connector.md
+++ b/docs/connector/connector.md
@@ -18,11 +18,11 @@ sources:
${prop_key}: ${prop_value}
```
-| Name | Type | Required | Default | Description |
-|-------------------------------------|------------------------------------------------------------------|----------|--------------------------|--------------------------------------------------------------------------------------------------------------------------|
-| type | String | Yes | - | The type of the source connector. The `SourceTableFactory` will use this value as identifier to create source connector. |
-| schema | Map | No | - | The source table schema, config through fields or local_file or url. |
-| properties | Map of String | Yes | - | The source connector customize properties, more details see the [Source](source) documentation. |
+| Name | Type | Required | Default | Description |
+|-------------|---------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------|
+| type | String | Yes | (none) | The type of the source connector. The `SourceTableFactory` will use this value as identifier to create source connector. |
+| schema | Map | No | (none) | The source table schema, config through fields or local_file or url. |
+| properties | Map of String | Yes | (none) | The source connector customize properties, more details see the [Source](source) documentation. |
## Schema Field Projection
The source connector supports reading only specified fields from the data source. For example `KafkaSource` will read all content from topic and then use `fields` to filter unnecessary columns.
@@ -85,9 +85,9 @@ sinks:
${prop_key}: ${prop_value}
```
-| Name | Type | Required | Default | Description |
-|-------------------------------------|--------------------------------------------------------------------|----------|--------------------------|--------------------------------------------------------------------------------------------------------------------|
-| type | String | Yes | - | The type of the sink connector. The `SinkTableFactory` will use this value as identifier to create sink connector. |
-| schema | Map | No | - | The sink table schema, config through fields or local_file or url. |
-| properties | Map of String | Yes | - | The sink connector customize properties, more details see the [Sink](sink) documentation. |
+| Name | Type | Required | Default | Description |
+|--------------|----------------|----------|---------|--------------------------------------------------------------------------------------------------------------------|
+| type | String | Yes | (none) | The type of the sink connector. The `SinkTableFactory` will use this value as identifier to create sink connector. |
+| schema | Map | No | (none) | The sink table schema, config through fields or local_file or url. |
+| properties | Map of String | Yes | (none) | The sink connector customize properties, more details see the [Sink](sink) documentation. |
diff --git a/docs/connector/formats/json.md b/docs/connector/formats/json.md
index 8756e89..01aecd7 100644
--- a/docs/connector/formats/json.md
+++ b/docs/connector/formats/json.md
@@ -9,10 +9,10 @@ Event serialization and deserialization format.
## Format Options
-| Name | Type | Required | Default | Description |
-|---------------------------|----------|----------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
-| format | String | Yes | - | Specify what format to use, here should be 'json'. |
-| json.ignore.parse.errors | Boolean | No | false | Skip the parsing error or throw an exception. If set to true, the parsing error will be ignored. If set to false, the parsing error will be thrown. |
+| Name | Type | Required | Default | Description |
+|---------------------------|----------|----------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
+| format | String | Yes | (none) | Specify what format to use, here should be 'json'. |
+| json.ignore.parse.errors | Boolean | No | false | Skip the parsing error or throw an exception. If set to true, the parsing error will be ignored. If set to false, the parsing error will be thrown. |
# How to use
## Inline uses example
diff --git a/docs/connector/formats/msgpack.md b/docs/connector/formats/msgpack.md
index 2184206..1b603a6 100644
--- a/docs/connector/formats/msgpack.md
+++ b/docs/connector/formats/msgpack.md
@@ -3,15 +3,15 @@
## Description
MessagePack is a binary serialization format. If you need a fast and compact alternative of JSON, MessagePack is your friend. For example, a small integer can be encoded in a single byte, and short strings only need a single byte prefix + the original byte array. MessagePack implementation is already available in various languages (See also the list in http://msgpack.org) and works as a universal data format.
-| Name | Supported Versions | Maven |
-|-------------|--------------------|----------------------------------------------------------------------------------------------------------------------------|
-| Format MessagePack | Universal | [Download](http://192.168.40.153:8099/service/local/repositories/platform-release/content/com/geedgenetworks/format-msgpack/) |
+| Name | Supported Versions | Maven |
+|---------------------|--------------------|-----------------------------------------------------------------------------------------------------------------------------------|
+| Format MessagePack | Universal | [Download](http://192.168.40.153:8099/service/local/repositories/platform-release/content/com/geedgenetworks/format-msgpack/) |
## Format Options
-| Name | Type | Required | Default | Description |
-|---------------------------|----------|----------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
-| format | String | Yes | - | Specify what format to use, here should be 'msgpack'. |
+| Name | Type | Required | Default | Description |
+|---------------------------|----------|----------|---------|---------------------------------------------------------|
+| format | String | Yes | (none) | Specify what format to use, here should be 'msgpack'. |
# How to use
## Inline uses example
diff --git a/docs/connector/formats/protobuf.md b/docs/connector/formats/protobuf.md
index 9d685be..e55f6f1 100644
--- a/docs/connector/formats/protobuf.md
+++ b/docs/connector/formats/protobuf.md
@@ -12,9 +12,9 @@ It is very popular in Streaming Data Pipeline. Now support protobuf format in so
| Name | Type | Required | Default | Description |
|-------------------------------|----------|----------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| format | String | Yes | - | Specify what format to use, here should be 'protobuf'. |
-| protobuf.descriptor.file.path | String | Yes | - | The descriptor file path. |
-| protobuf.message.name | String | Yes | - | The protobuf messageName to look for in descriptor file. |
+| format | String | Yes | (none) | Specify what format to use, here should be 'protobuf'. |
+| protobuf.descriptor.file.path | String | Yes | (none) | The descriptor file path. |
+| protobuf.message.name | String | Yes | (none) | The protobuf messageName to look for in descriptor file. |
| protobuf.ignore.parse.errors | Boolean | No | false | Protobuf ignore parse errors, otherwise will throw exception. |
| protobuf.emit.default.values | Boolean | No | false | If true, default values will be emitted for missing fields. It is not recommended, because it will cause performance degradation. About basic data type, it is suggested to use `optional` instead of `required`. |
diff --git a/docs/connector/formats/raw.md b/docs/connector/formats/raw.md
index 0f7e53f..ae91fe9 100644
--- a/docs/connector/formats/raw.md
+++ b/docs/connector/formats/raw.md
@@ -11,7 +11,7 @@ The Raw format allows to read and write raw (byte based) values as a single colu
| Name | Type | Required | Default | Description |
|---------------------------|----------|----------|---------|---------------------------------------------------|
-| format | String | Yes | - | Specify what format to use, here should be 'raw'. |
+| format | String | Yes | (none) | Specify what format to use, here should be 'raw'. |
# How to use
## Inline uses example
diff --git a/docs/connector/sink/clickhouse.md b/docs/connector/sink/clickhouse.md
index ac37d24..c11ecbd 100644
--- a/docs/connector/sink/clickhouse.md
+++ b/docs/connector/sink/clickhouse.md
@@ -6,6 +6,10 @@ Sink connector for ClickHouse, write data to ClickHouse. You need to know follow
> 2. The Sink Table must be created before using the ClickHouse connector.
> 3. When writing data to sink table, you don't need to specify its schema, because the connector will automatically query current table's schema information using `DESCRIBE TABLE` command.
+## Key Features
+
+- [ ] [Rate Limiting](../../connector/connector.md)
+
## Data Type Mapping
@@ -21,6 +25,8 @@ Sink connector for ClickHouse, write data to ClickHouse. You need to know follow
| TIME | DateTime |
| ARRAY | Array |
| MAP | Map |
+| Byte[] | Binary |
+
## Sink Options
In order to use the ClickHouse connector, the following dependencies are required. They can be download by Nexus Maven Repository.
@@ -34,14 +40,14 @@ ClickHouse sink custom properties. If properties belongs to ClickHouse JDBC Conf
| Name | Type | Required | Default | Description |
|---------------------|----------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| host | String | Yes | - | `ClickHouse` cluster address, the format is `host:port` , allowing multiple `hosts` to be specified. Such as `"host1:8123,host2:8123"`. |
-| connection.database | String | Yes | - | The `ClickHouse` database. |
-| table | String | Yes | - | The table name. |
+| host | String | Yes | (none) | `ClickHouse` cluster address, the format is `host:port` , allowing multiple `hosts` to be specified. Such as `"host1:8123,host2:8123"`. |
+| connection.database | String | Yes | (none) | The `ClickHouse` database. |
+| table | String | Yes | (none) | The table name. |
| batch.size | Integer | Yes | 100000 | The number of rows written through [Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) each time, the `default is 20000`. |
| batch.interval | Duration | Yes | 30s | The time interval for writing data through. |
-| connection.user | String | Yes | - | The username to use to connect to `ClickHouse`. |
-| connection.password | String | Yes | - | The password to use to connect to `ClickHouse`. |
-| connection.config | Map | No | - | In addition to the above mandatory parameters that must be specified by `clickhouse-jdbc` , users can also specify multiple optional parameters, which cover all the [parameters](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration) provided by `clickhouse-jdbc`. |
+| connection.user | String | Yes | (none) | The username to use to connect to `ClickHouse`. |
+| connection.password | String | Yes | (none) | The password to use to connect to `ClickHouse`. |
+| connection.config | Map | No | (none) | In addition to the above mandatory parameters that must be specified by `clickhouse-jdbc` , users can also specify multiple optional parameters, which cover all the [parameters](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration) provided by `clickhouse-jdbc`. |
## Example
This example read data of inline test source and write to ClickHouse table `test`.
diff --git a/docs/connector/sink/kafka.md b/docs/connector/sink/kafka.md
index 92976d8..e4d1a3f 100644
--- a/docs/connector/sink/kafka.md
+++ b/docs/connector/sink/kafka.md
@@ -2,6 +2,11 @@
> Kafka sink connector
## Description
Sink connector for Apache Kafka. Write data to Kafka topic.
+
+## Key Features
+
+- [x] [Rate Limiting](../../connector/connector.md)
+
## Sink Options
In order to use the Kafka connector, the following dependencies are required. They can be download by Nexus Maven Repository.
@@ -11,14 +16,18 @@ In order to use the Kafka connector, the following dependencies are required. Th
Kafka sink custom properties. if properties belongs to Kafka Producer Config, you can use `kafka.` prefix to set.
-| Name | Type | Required | Default | Description |
-|-------------------------|---------|----------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| topic | String | Yes | - | Topic name is required. It used to write data to kafka. |
-| kafka.bootstrap.servers | String | Yes | - | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. This list should be in the form `host1:port1,host2:port2,...`. |
-| log.failures.only | Boolean | No | true | Defines whether the producer should fail on errors, or only log them. If this is set to true, then exceptions will be only logged, if set to false, exceptions will be eventually thrown and cause the streaming program to fail (and enter recovery). |
-| format | String | No | json | Data format. The default value is `json`. The Optional values are `json`, `protobuf`. |
-| [format].config | | No | - | Data format properties. Please refer to [Format Options](../formats) for details. |
-| kafka.config | | No | - | Kafka producer properties. Please refer to [Kafka Producer Config](https://kafka.apache.org/documentation/#producerconfigs) for details. |
+| Name | Type | Required | Default | Description |
+|-------------------------------------|-----------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| topic | String | Yes | (none) | Topic name is required. It used to write data to kafka. |
+| kafka.bootstrap.servers | String | Yes | (none) | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. This list should be in the form `host1:port1,host2:port2,...`. |
+| log.failures.only | Boolean | No | true | Defines whether the producer should fail on errors, or only log them. If this is set to true, then exceptions will be only logged, if set to false, exceptions will be eventually thrown and cause the streaming program to fail (and enter recovery). |
+| format | String | No | json | Data format. The default value is `json`. The Optional values are `json`, `protobuf`. |
+| [format].config | / | No | (none) | Data format properties. Please refer to [Format Options](../formats) for details. |
+| rate.limiting.strategy | String | No | (none) | The rate limiting strategy to use. The Optional values are `none`, `sliding_window`. |
+| rate.limiting.limit.rate | String | No | 10Mbps | A maximum rate of traffic that can be transmitted over a network or between networks. The units of the bytes rate are Mbps, Kbps,and bps. For example, 10Mbps, 100Kbps, 1000bps. |
+| rate.limiting.block.duration | String | No | 5min | If the rate limit is exceeded, the data will be blocked for the specified duration. The units of the duration are seconds, minutes, and hours. For example, 10s, 1m, 1h. |
+| rate.limiting.block.reset.duration | String | No | 30s | The time interval for resetting the rate limit. The units of the duration are seconds, minutes, and hours. For example, 10s, 1m, 1h. |
+| kafka.config | / | No | (none) | Kafka producer properties. Please refer to [Kafka Producer Config](https://kafka.apache.org/documentation/#producerconfigs) for details. |
## Example
This example read data of inline test source and write to kafka topic `SESSION-RECORD-TEST`.
diff --git a/docs/connector/sink/print.md b/docs/connector/sink/print.md
index 271d7a2..b76379d 100644
--- a/docs/connector/sink/print.md
+++ b/docs/connector/sink/print.md
@@ -8,8 +8,8 @@ Print sink custom properties.
| Name | Type | Required | Default | Description |
|-----------------|----------|----------|---------|-----------------------------------------------------------------------------------|
-| format | String | Yes | - | Data format. The Optional values are `json`, `protobuf`. |
-| [format].config | Map | No | - | Data format properties. Please refer to [Format Options](../formats) for details. |
+| format | String | Yes | (none) | Data format. The Optional values are `json`, `protobuf`. |
+| [format].config | Map | No | (none) | Data format properties. Please refer to [Format Options](../formats) for details. |
| mode | String | No | stdout | Print mode. The Optional values are `stdout`, `log_info`, `log_warn`, `null`. |
## Example
diff --git a/docs/connector/source/inline.md b/docs/connector/source/inline.md
index c91d1a7..b510c20 100644
--- a/docs/connector/source/inline.md
+++ b/docs/connector/source/inline.md
@@ -7,9 +7,9 @@ Inline source custom properties. The data will be parsed to Map<String, Object>
| Name | Type | Required | Default | Description |
|------------------|-----------|----------|---------|--------------------------------------------------------------------------------------------------------------------------|
-| data | String | Yes | - | Testing data, support json and protobuf format. If data is json format , it can be parsed multiple events by json array. |
-| format | String | Yes | - | Data format. The Optional values are `json`, `protobuf`. |
-| [format].config | Map | No | - | Data format properties. Please refer to [Format Options](../formats) for details. |
+| data | String | Yes | (none) | Testing data, support json and protobuf format. If data is json format , it can be parsed multiple events by json array. |
+| format | String | Yes | (none) | Data format. The Optional values are `json`, `protobuf`. |
+| [format].config | Map | No | (none) | Data format properties. Please refer to [Format Options](../formats) for details. |
| interval.per.row | Duration | No | 1s | Interval time in seconds between each row. If it is 0, it will be generated as fast as possible. |
| repeat.count | Integer | No | -1 | Repeat count. If it is -1, it will be generated infinitely. |
| type | String | No | string | Data codec type. The Optional values are `string(UTF-8)`, `hex`, `base64`. |
diff --git a/docs/connector/source/ipfix.md b/docs/connector/source/ipfix.md
index c70078e..3a4fbbf 100644
--- a/docs/connector/source/ipfix.md
+++ b/docs/connector/source/ipfix.md
@@ -13,19 +13,19 @@ IPFIX source custom properties.
| Name | Type | Required | Default | Description |
|-----------------------------------------|---------|----------|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| port.range | String | Yes | - | Range of ports such as 3000 -3010 |
+| port.range | String | Yes | (none) | Range of ports such as 3000 -3010 |
| max.packet.size | Integer | No | 65535 | The maximum size of packet for UDP |
| max.receive.buffer.size | Integer | No | 104857600 | UDP maximum receive buffer size in bytes |
-| service.discovery.registry.mode | String | No | - | Service discovery registry mode, support `0(nacos)` and `1(consul)` |
-| service.discovery.service.name | String | No | - | Service discovery service name |
-| service.discovery.health.check.interval | Integer | No | - | Service discovery health check interval in milliseconds |
-| service.discovery.nacos.server.addr | String | No | - | A list of host/port pairs to use for establishing the initial connection to the Nacos cluster. This list should be in the form `host1:port1,host2:port2,...`. |
-| service.discovery.nacos.username | String | No | - | The username to use to connect to Nacos.Need specify `ipfix.service.discovery.registry.mode` to `0(nacos)` |
-| service.discovery.nacos.password | String | No | - | The password to use to connect to Nacos.Need specify `ipfix.service.discovery.registry.mode` to `0(nacos)` |
-| service.discovery.nacos.namespace | String | No | - | The namespace to use to connect to Nacos.Need specify `ipfix.service.discovery.registry.mode` to `0(nacos)` |
-| service.discovery.nacos.group | String | No | - | The group to use to connect to Nacos.Need specify `ipfix.service.discovery.registry.mode` to `0(nacos)` |
-| service.discovery.consul.server.ip | String | No | - | Consul server IP address. Need specify `ipfix.service.discovery.registry.mode` to `1(consul)` |
-| service.discovery.consul.server.port | Integer | No | - | Consul server port. Need specify `ipfix.service.discovery.registry.mode` to `1(consul)` |
-| service.discovery.consul.token | String | No | - | Consul token. Need specify `ipfix.service.discovery.registry.mode` to `1(consul)` |
+| service.discovery.registry.mode | String | No | (none) | Service discovery registry mode, support `0(nacos)` and `1(consul)` |
+| service.discovery.service.name | String | No | (none) | Service discovery service name |
+| service.discovery.health.check.interval | Integer | No | (none) | Service discovery health check interval in milliseconds |
+| service.discovery.nacos.server.addr | String | No | (none) | A list of host/port pairs to use for establishing the initial connection to the Nacos cluster. This list should be in the form `host1:port1,host2:port2,...`. |
+| service.discovery.nacos.username | String | No | (none) | The username to use to connect to Nacos.Need specify `ipfix.service.discovery.registry.mode` to `0(nacos)` |
+| service.discovery.nacos.password | String | No | (none) | The password to use to connect to Nacos.Need specify `ipfix.service.discovery.registry.mode` to `0(nacos)` |
+| service.discovery.nacos.namespace | String | No | (none) | The namespace to use to connect to Nacos.Need specify `ipfix.service.discovery.registry.mode` to `0(nacos)` |
+| service.discovery.nacos.group | String | No | (none) | The group to use to connect to Nacos.Need specify `ipfix.service.discovery.registry.mode` to `0(nacos)` |
+| service.discovery.consul.server.ip | String | No | (none) | Consul server IP address. Need specify `ipfix.service.discovery.registry.mode` to `1(consul)` |
+| service.discovery.consul.server.port | Integer | No | (none) | Consul server port. Need specify `ipfix.service.discovery.registry.mode` to `1(consul)` |
+| service.discovery.consul.token | String | No | (none) | Consul token. Need specify `ipfix.service.discovery.registry.mode` to `1(consul)` |
## Example \ No newline at end of file
diff --git a/docs/connector/source/kafka.md b/docs/connector/source/kafka.md
index 0565fd4..52d541b 100644
--- a/docs/connector/source/kafka.md
+++ b/docs/connector/source/kafka.md
@@ -2,6 +2,7 @@
> Kafka source connector
## Description
Source connector for Apache Kafka
+
## Source Options
In order to use the Kafka connector, the following dependencies are required. They can be download by Nexus Maven Repository.
@@ -11,13 +12,13 @@ In order to use the Kafka connector, the following dependencies are required. Th
Kafka source custom properties. if properties belongs to Kafka Consumer Config, you can use `kafka.` prefix to set.
-| Name | Type | Required | Default | Description |
-|-------------------------|--------|----------|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| topic | String | Yes | - | Topic name(s). It also supports topic list for source by separating topic by semicolon like 'topic-1;topic-2'. |
-| kafka.bootstrap.servers | String | Yes | - | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. This list should be in the form `host1:port1,host2:port2,...`. |
-| format | String | No | json | Data format. The default value is `json`. The Optional values are `json`, `protobuf`. |
-| [format].config | Map | No | - | Data format properties. Please refer to [Format Options](../formats) for details. |
-| kafka.config | Map | No | - | Kafka consumer properties. Please refer to [Kafka Consumer Config](https://kafka.apache.org/documentation/#consumerconfigs) for details. |
+| Name | Type | Required | Default | Description |
+|-------------------------|--------|----------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| topic | String | Yes | (none) | Topic name(s). It also supports topic list for source by separating topic by semicolon like 'topic-1;topic-2'. |
+| kafka.bootstrap.servers | String | Yes | (none) | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. This list should be in the form `host1:port1,host2:port2,...`. |
+| format | String | No | json | Data format. The default value is `json`. The Optional values are `json`, `protobuf`. |
+| [format].config | Map | No | (none) | Data format properties. Please refer to [Format Options](../formats) for details. |
+| kafka.config | Map | No | (none) | Kafka consumer properties. Please refer to [Kafka Consumer Config](https://kafka.apache.org/documentation/#consumerconfigs) for details. |
## Example
This example read data of kafka topic `SESSION-RECORD` and print to console.
generated by cgit v1.2.3 (git 2.39.1) at 2025-12-30 17:03:46 +0000