summaryrefslogtreecommitdiff
path: root/docs/connector
diff options
context:
space:
mode:
authorlifengchao <[email protected]>2024-10-31 14:39:19 +0800
committerlifengchao <[email protected]>2024-10-31 14:39:19 +0800
commitfa5729fd4a2292400be61fdfc2e7f6719928c87b (patch)
treeae10dd7d9db994d0bb6a7add69e18e72074147e1 /docs/connector
parent06975ee829f9395f095a12c10eaedffcd89b3d83 (diff)
[feature][format-csv]GAL-687 Groot Stream 支持CSV Format
Diffstat (limited to 'docs/connector')
-rw-r--r--docs/connector/formats/csv.md73
-rw-r--r--docs/connector/formats/raw.md2
2 files changed, 74 insertions, 1 deletions
diff --git a/docs/connector/formats/csv.md b/docs/connector/formats/csv.md
new file mode 100644
index 0000000..ca8d10b
--- /dev/null
+++ b/docs/connector/formats/csv.md
@@ -0,0 +1,73 @@
+# CSV
+
+> Format CSV
+>
+> ## Description
+>
+> The CSV format allows to read and write CSV data based on an CSV schema. Currently, the CSV schema is derived from table schema.
+> **The CSV format must config schema for source/sink**.
+
+| Name | Supported Versions | Maven |
+|--------------|--------------------|---------------------------------------------------------------------------------------------------------------------------|
+| Format CSV | Universal | [Download](http://192.168.40.153:8099/service/local/repositories/platform-release/content/com/geedgenetworks/format-csv/) |
+
+## Format Options
+
+| Name | Type | Required | Default | Description |
+|-----------------------------|-----------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| format | String | Yes | (none) | Specify what format to use, here should be 'csv'. |
+| csv.field.delimiter | String | No | , | Field delimiter character (',' by default), must be single character. You can use backslash to specify special characters, e.g. '\t' represents the tab character. |
+| csv.disable.quote.character | Boolean | No | false | Disabled quote character for enclosing field values (false by default). If true, option 'csv.quote.character' can not be set. |
+| csv.quote.character | String | No | " | Quote character for enclosing field values (" by default). |
+| csv.allow.comments | Boolean | No | false | Ignore comment lines that start with '#' (disabled by default). If enabled, make sure to also ignore parse errors to allow empty rows. |
+| csv.ignore.parse.errors | Boolean | No | false | Skip fields and rows with parse errors instead of failing. Fields are set to null in case of errors. |
+| csv.array.element.delimiter | String | No | ; | Array element delimiter string for separating array and row element values (';' by default). |
+| csv.escape.character | String | No | (none) | Escape character for escaping values (disabled by default). |
+| csv.null.literal | String | No | (none) | Null literal string that is interpreted as a null value (disabled by default). |
+
+# How to use
+
+## Inline uses example
+
+data:
+
+```json
+{
+ "log_id": 1,
+ "recv_time": 1712827485,
+ "client_ip": "192.168.0.1"
+}
+```
+
+```yaml
+sources:
+ inline_source:
+ type: inline
+ schema:
+ fields: "log_id:int, recv_time:bigint, client_ip:string"
+ properties:
+ data: "1,1712827485,192.168.0.1"
+ format: csv
+
+sinks:
+ print_sink:
+ type: print
+ schema:
+ fields: "log_id:int, recv_time:bigint, client_ip:string"
+ properties:
+ format: csv
+
+application:
+ env:
+ name: example-inline-to-print
+ parallelism: 3
+ pipeline:
+ object-reuse: true
+ topology:
+ - name: inline_source
+ downstream: [print_sink]
+ - name: print_sink
+ downstream: []
+
+```
+
diff --git a/docs/connector/formats/raw.md b/docs/connector/formats/raw.md
index 853ac79..06ea8bc 100644
--- a/docs/connector/formats/raw.md
+++ b/docs/connector/formats/raw.md
@@ -4,7 +4,7 @@
>
> ## Description
>
-> The Raw format allows to read and write raw (byte based) values as a single column.
+> The Raw format allows to read and write raw (byte based) values as a single column, the column name is raw default, it can also be explicitly defined as other name.
| Name | Supported Versions | Maven |
|------------|--------------------|---------------------------------------------------------------------------------------------------------------------------|