[Improve][docs] Add ClickHouse.md and improve Kafka connector details.

3 files changed, 198 insertions, 10 deletions

diff --git a/docs/connector/sink/ClickHouse.md b/docs/connector/sink/ClickHouse.md
index e69de29..21d5de9 100644
--- a/docs/connector/sink/ClickHouse.md
+++ b/docs/connector/sink/ClickHouse.md
@@ -0,0 +1,105 @@
+# ClickHouse
+> ClickHouse sink connector
+## Description
+Sink connector for ClickHouse, write data to ClickHouse. You need to know following concepts before using ClickHouse connector.
+> 1. ClickHouse is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP).
+> 2. The Sink Table must be created before using the ClickHouse connector.
+> 3. When writing data to sink table, you don't need to specify its schema, because the connector will automatically query current table's schema information using `DESCRIBE TABLE` command.
+
+
+## Data Type Mapping
+
+| Event Data Type | Clickhouse Data Type                                                               |
+|-----------------|------------------------------------------------------------------------------------|
+| STRING          | String / Int128 / UInt128 / Int256 / UInt256 / Point / Ring / Polygon MultiPolygon |
+| INT             | Int8 / UInt8 / Int16 / UInt16 / Int32                                              |
+| BIGINT          | UInt64 / Int64                                                                     |
+| DOUBLE          | Float64                                                                            |
+| DECIMAL         | Decimal                                                                            |
+| FLOAT           | Float32                                                                            |
+| DATE            | Date                                                                               |
+| TIME            | DateTime                                                                           |
+| ARRAY           | Array                                                                              |
+| MAP             | Map                                                                                |
+
+## Sink Options
+In order to use the ClickHouse connector, the following dependencies are required. They can be download by Nexus Maven Repository.
+
+| Datasource | Supported Versions | Maven                                                                                                                               |
+|------------|--------------------|-------------------------------------------------------------------------------------------------------------------------------------|
+| ClickHouse | Universal          | [Download](http://192.168.40.153:8099/service/local/repositories/platform-release/content/com/geedgenetworks/connector-clickhouse/) |
+
+
+ClickHouse sink customizes properties. if properties belongs to ClickHouse JDBC Config, you can use `connection.` prefix to set.
+
+| Name                  | Type     | Required | Default | Description                                                                                                                                                                                                                                                                                                  |
+|-----------------------|----------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| host                  | String   | Yes      | -       | `ClickHouse` cluster address, the format is `host:port` , allowing multiple `hosts` to be specified. Such as `"host1:8123,host2:8123"`.                                                                                                                                                                      |
+| database              | String   | Yes      | -       | The `ClickHouse` database.                                                                                                                                                                                                                                                                                   |
+| table                 | String   | Yes      | -       | The table name.                                                                                                                                                                                                                                                                                              |
+| batch.size            | Integer  | Yes      | 100000  | The number of rows written through [Clickhouse-jdbc](https://github.com/ClickHouse/clickhouse-jdbc) each time, the `default is 20000`.                                                                                                                                                                       |
+| batch.interval        | Duration | Yes      | 30s     | The time interval for writing data through.                                                                                                                                                                                                                                                                  |
+| connection.user       | String   | Yes      | -       | The username to use to connect to `ClickHouse`.                                                                                                                                                                                                                                                              |
+| connection.password   | String   | Yes      | -       | The password to use to connect to `ClickHouse`.                                                                                                                                                                                                                                                              |
+| connection.config     | Map      | No       | -       | In addition to the above mandatory parameters that must be specified by `clickhouse-jdbc` , users can also specify multiple optional parameters, which cover all the [parameters](https://github.com/ClickHouse/clickhouse-jdbc/tree/master/clickhouse-client#configuration) provided by `clickhouse-jdbc`.  |
+
+## Example
+This example read data of inline test source and write to ClickHouse table `test`.
+
+```yaml
+sources: # [object] Define connector source
+  inline_source:
+    type: inline
+    fields: # [array of object] Schema field projection, support read data only from specified fields.
+      - name: log_id
+        type: bigint
+      - name: recv_time
+        type: bigint
+      - name: server_fqdn
+        type: string
+      - name: server_domain
+        type: string
+      - name: client_ip
+        type: string
+      - name: server_ip
+        type: string
+      - name: server_asn
+        type: string
+      - name: decoded_as
+        type: string
+      - name: device_group
+        type: string
+      - name: device_tag
+        type: string
+    properties:
+      #
+      # [string] Event Data, it will be parsed to Map<String, Object> by the specified format.
+      #
+      data: '{"recv_time": 1705565615, "log_id":206211012872372224, "tcp_rtt_ms":128,"decoded_as":"HTTP", "http_version":"http1","http_request_line":"GET / HTTP/1.1","http_host":"www.ct.cn","http_url":"www.ct.cn/","http_user_agent":"curl/8.0.1","http_status_code":200,"http_response_line":"HTTP/1.1 200 OK","http_response_content_type":"text/html; charset=UTF-8","http_response_latency_ms":31,"http_session_duration_ms":5451,"in_src_mac":"ba:bb:a7:3c:67:1c","in_dest_mac":"86:dd:7a:8f:ae:e2","out_src_mac":"86:dd:7a:8f:ae:e2","out_dest_mac":"ba:bb:a7:3c:67:1c","tcp_client_isn":678677906,"tcp_server_isn":1006700307,"address_type":4,"client_ip":"192.11.22.22","server_ip":"8.8.8.8","client_port":42751,"server_port":80,"in_link_id":65535,"out_link_id":65535,"start_timestamp_ms":1703646546127,"end_timestamp_ms":1703646551702,"duration_ms":5575,"sent_pkts":97,"sent_bytes":5892,"received_pkts":250,"received_bytes":333931,"tcp_c2s_ip_fragments":0,"tcp_s2c_ip_fragments":0,"tcp_c2s_rtx_pkts":0,"tcp_c2s_rtx_bytes":0,"tcp_s2c_rtx_pkts":0,"tcp_s2c_rtx_bytes":0,"tcp_c2s_o3_pkts":0,"tcp_s2c_o3_pkts":0,"tcp_c2s_lost_bytes":0,"tcp_s2c_lost_bytes":0,"flags":26418,"flags_identify_info":[100,1,100,60,150,100,1,2],"app_transition":"http.1111.test_1_1","decoded_as":"HTTP","server_fqdn":"www.ct.cn","app":"test_1_1","decoded_path":"ETHERNET.IPv4.TCP.http","fqdn_category_list":[1767],"t_vsys_id":1,"vsys_id":1,"session_id":290538039798223400,"tcp_handshake_latency_ms":41,"client_os_desc":"Windows","server_os_desc":"Linux","data_center":"center-xxg-tsgx","device_group":"group-xxg-tsgx","device_tag":"{\"tags\":[{\"tag\":\"data_center\",\"value\":\"center-xxg-tsgx\"},{\"tag\":\"device_group\",\"value\":\"group-xxg-tsgx\"}]}","device_id":"9800165603247024","sled_ip":"192.168.40.39","dup_traffic_flag":0}'
+      format: json
+      json.ignore.parse.errors: false
+
+sinks:
+  clickhouse_sink:
+    type: clickhouse
+    properties:
+      host: 192.168.44.12:9001
+      table: tsg_galaxy_v3.inline_source_test_local
+      batch.size: 10
+      batch.interval: 1s
+      connection.user: *
+      connection.password: *
+
+application:  # [object] Define job configuration
+  env:
+    name: groot-stream-job-inline-to-kafka
+    parallelism: 3
+    pipeline:
+      object-reuse: true
+  topology:
+    - name: inline_source
+      downstream: [clickhouse_sink]
+    - name: clickhouse_sink
+      downstream: []
+```
+
diff --git a/docs/connector/sink/Kafka.md b/docs/connector/sink/Kafka.md
index e69de29..4073912 100644
--- a/docs/connector/sink/Kafka.md
+++ b/docs/connector/sink/Kafka.md
@@ -0,0 +1,83 @@
+# Kafka
+> Kafka sink connector
+## Description
+Sink connector for Apache Kafka. Write data to Kafka topic.
+## Sink Options
+In order to use the Kafka connector, the following dependencies are required. They can be download by Nexus Maven Repository.
+
+| Datasource | Supported Versions | Maven                                                                                                                          |
+|------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------|
+| Kafka      | Universal          | [Download](http://192.168.40.153:8099/service/local/repositories/platform-release/content/com/geedgenetworks/connector-kafka/) |
+
+Kafka sink customizes properties. if properties belongs to Kafka Producer Config, you can use `kafka.` prefix to set.
+
+| Name                    | Type   | Required | Default | Description                                                                                                                                                   |
+|-------------------------|--------|----------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| topic                   | String | Yes      | -       | Topic name is required. It used to write data to kafka.                                                                                                       |                                             
+| kafka.bootstrap.servers | String | Yes      | -       | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. This list should be in the form `host1:port1,host2:port2,...`. |
+| format                  | String | No       | json    | Data format. The default value is `json`. The Optional values are `json`, `protobuf`.                                                                         |
+| [format].config         |        | No       | -       | Data format properties. Please refer to [Format Options](../formats) for details.                                                                             |                                                                                                                                       
+| kafka.config            |        | No       | -       | Kafka producer properties. Please refer to [Kafka Producer Config](https://kafka.apache.org/documentation/#producerconfigs) for details.                      |
+
+## Example
+This example read data of inline test source and write to kafka topic `SESSION-RECORD-TEST`.
+```yaml
+sources: # [object] Define connector source
+  inline_source:
+    type: inline
+    fields: # [array of object] Schema field projection, support read data only from specified fields.
+      - name: log_id
+        type: bigint
+      - name: recv_time
+        type: bigint
+      - name: server_fqdn
+        type: string
+      - name: server_domain
+        type: string
+      - name: client_ip
+        type: string
+      - name: server_ip
+        type: string
+      - name: server_asn
+        type: string
+      - name: decoded_as
+        type: string
+      - name: device_group
+        type: string
+      - name: device_tag
+        type: string
+    properties:
+      #
+      # [string] Event Data, it will be parsed to Map<String, Object> by the specified format.
+      #
+      data: '{"tcp_rtt_ms":128,"decoded_as":"HTTP", "http_version":"http1","http_request_line":"GET / HTTP/1.1","http_host":"www.ct.cn","http_url":"www.ct.cn/","http_user_agent":"curl/8.0.1","http_status_code":200,"http_response_line":"HTTP/1.1 200 OK","http_response_content_type":"text/html; charset=UTF-8","http_response_latency_ms":31,"http_session_duration_ms":5451,"in_src_mac":"ba:bb:a7:3c:67:1c","in_dest_mac":"86:dd:7a:8f:ae:e2","out_src_mac":"86:dd:7a:8f:ae:e2","out_dest_mac":"ba:bb:a7:3c:67:1c","tcp_client_isn":678677906,"tcp_server_isn":1006700307,"address_type":4,"client_ip":"192.11.22.22","server_ip":"8.8.8.8","client_port":42751,"server_port":80,"in_link_id":65535,"out_link_id":65535,"start_timestamp_ms":1703646546127,"end_timestamp_ms":1703646551702,"duration_ms":5575,"sent_pkts":97,"sent_bytes":5892,"received_pkts":250,"received_bytes":333931,"tcp_c2s_ip_fragments":0,"tcp_s2c_ip_fragments":0,"tcp_c2s_rtx_pkts":0,"tcp_c2s_rtx_bytes":0,"tcp_s2c_rtx_pkts":0,"tcp_s2c_rtx_bytes":0,"tcp_c2s_o3_pkts":0,"tcp_s2c_o3_pkts":0,"tcp_c2s_lost_bytes":0,"tcp_s2c_lost_bytes":0,"flags":26418,"flags_identify_info":[100,1,100,60,150,100,1,2],"app_transition":"http.1111.test_1_1","decoded_as":"HTTP","server_fqdn":"www.ct.cn","app":"test_1_1","decoded_path":"ETHERNET.IPv4.TCP.http","fqdn_category_list":[1767],"t_vsys_id":1,"vsys_id":1,"session_id":290538039798223400,"tcp_handshake_latency_ms":41,"client_os_desc":"Windows","server_os_desc":"Linux","data_center":"center-xxg-tsgx","device_group":"group-xxg-tsgx","device_tag":"{\"tags\":[{\"tag\":\"data_center\",\"value\":\"center-xxg-tsgx\"},{\"tag\":\"device_group\",\"value\":\"group-xxg-tsgx\"}]}","device_id":"9800165603247024","sled_ip":"192.168.40.39","dup_traffic_flag":0}'
+      format: json
+      json.ignore.parse.errors: false
+
+sinks:
+  connector_kafka:
+    type: kafka
+    properties:
+      topic: SESSION-RECORD-TEST
+      kafka.bootstrap.servers: 192.168.44.12:9092
+      kafka.retries: 0
+      kafka.linger.ms: 10
+      kafka.request.timeout.ms: 30000
+      kafka.batch.size: 262144
+      kafka.buffer.memory: 134217728
+      kafka.max.request.size: 10485760
+      kafka.compression.type: snappy
+      format: json
+
+application:  # [object] Define job configuration
+  env:
+    name: groot-stream-job-inline-to-kafka
+    parallelism: 3
+    pipeline:
+      object-reuse: true
+  topology:
+    - name: inline_source
+      downstream: [connector_kafka]
+    - name: connector_kafka
+      downstream: []
+```
diff --git a/docs/connector/source/Kafka.md b/docs/connector/source/Kafka.md
index 49ea262..4d9b34d 100644
--- a/docs/connector/source/Kafka.md
+++ b/docs/connector/source/Kafka.md
@@ -5,19 +5,19 @@ Source connector for Apache Kafka
 ## Source Options
 In order to use the Kafka connector, the following dependencies are required. They can be download by Nexus Maven Repository.
 
-| Datasource      | Supported Versions | Maven                                                                                                                          |
-|-----------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------|
-| connector-kafka | Universal          | [Download](http://192.168.40.153:8099/service/local/repositories/platform-release/content/com/geedgenetworks/connector-kafka/) |
+| Datasource | Supported Versions | Maven                                                                                                                          |
+|------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------|
+| Kafka      | Universal          | [Download](http://192.168.40.153:8099/service/local/repositories/platform-release/content/com/geedgenetworks/connector-kafka/) |
 
 Kafka source customizes properties. if properties belongs to Kafka Consumer Config, you can use `kafka.` prefix to set.
 
-| Name                      | Type   | Required | Default      | Description                                                                                                                                                   |
-|---------------------------|--------|----------|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| topic                     | String | Yes      | -            | Topic name(s). It also supports topic list for source by using comma to separate topic names. eg. `topic1,topic2`                                             |
-| kafka.bootstrap.servers   | String | Yes      | -            | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. This list should be in the form `host1:port1,host2:port2,...`. |
-| format                    | String | No       | json         | Data format. The default value is `json`. The Optional values are `json`, `protobuf`.                                                                         |
-| Format Properties         |        | No       | -            | Data format properties. Please refer to [Format Options](../formats) for details.                                                                             |                                                                                                                                       
-| Kafka Consumer Properties |        | No       | -            | Kafka consumer properties. Please refer to [Kafka Consumer Config](https://kafka.apache.org/documentation/#consumerconfigs) for details.                      |
+| Name                    | Type   | Required | Default      | Description                                                                                                                                                   |
+|-------------------------|--------|----------|--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| topic                   | String | Yes      | -            | Topic name(s). It also supports topic list for source by separating topic by semicolon like 'topic-1;topic-2'.                                                |
+| kafka.bootstrap.servers | String | Yes      | -            | A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. This list should be in the form `host1:port1,host2:port2,...`. |
+| format                  | String | No       | json         | Data format. The default value is `json`. The Optional values are `json`, `protobuf`.                                                                         |
+| [format].config         | Map    | No       | -            | Data format properties. Please refer to [Format Options](../formats) for details.                                                                             |                                                                                                                                       
+| kafka.config            | Map    | No       | -            | Kafka consumer properties. Please refer to [Kafka Consumer Config](https://kafka.apache.org/documentation/#consumerconfigs) for details.                      |
 
 ## Example
 This example read data of kafka topic `SESSION-RECORD` and print to console.