Merge branch 'develop' into 'master'master

Revert "[Fix][schema] 修正dos event schema field_discovery_metric session计算方式(TSG-21222)"

See merge request galaxy/platform/galaxy-qgw-service!40

43 files changed, 136 insertions, 2 deletions

diff --git a/docs/release/release-340.md b/docs/release/2021/release-340.md
index bdd00e07..bdd00e07 100644
--- a/docs/release/release-340.md
+++ b/docs/release/2021/release-340.md
diff --git a/docs/release/release-341.md b/docs/release/2021/release-341.md
index e02967a0..e02967a0 100644
--- a/docs/release/release-341.md
+++ b/docs/release/2021/release-341.md
diff --git a/docs/release/release-342.md b/docs/release/2021/release-342.md
index ff4621d7..ff4621d7 100644
--- a/docs/release/release-342.md
+++ b/docs/release/2021/release-342.md
diff --git a/docs/release/release-343.md b/docs/release/2021/release-343.md
index 58cda80d..58cda80d 100644
--- a/docs/release/release-343.md
+++ b/docs/release/2021/release-343.md
diff --git a/docs/release/release-344.md b/docs/release/2021/release-344.md
index 74bed113..74bed113 100644
--- a/docs/release/release-344.md
+++ b/docs/release/2021/release-344.md
diff --git a/docs/release/release-345.md b/docs/release/2021/release-345.md
index d532596e..d532596e 100644
--- a/docs/release/release-345.md
+++ b/docs/release/2021/release-345.md
diff --git a/docs/release/release-346.md b/docs/release/2021/release-346.md
index db9bc831..db9bc831 100644
--- a/docs/release/release-346.md
+++ b/docs/release/2021/release-346.md
diff --git a/docs/release/release-347.md b/docs/release/2021/release-347.md
index 5838d4d5..5838d4d5 100644
--- a/docs/release/release-347.md
+++ b/docs/release/2021/release-347.md
diff --git a/docs/release/release-348.md b/docs/release/2021/release-348.md
index aa735cb0..aa735cb0 100644
--- a/docs/release/release-348.md
+++ b/docs/release/2021/release-348.md
diff --git a/docs/release/release-349.md b/docs/release/2021/release-349.md
index 88e5ce7f..88e5ce7f 100644
--- a/docs/release/release-349.md
+++ b/docs/release/2021/release-349.md
diff --git a/docs/release/release-350.md b/docs/release/2021/release-350.md
index 4f8cd4cf..4f8cd4cf 100644
--- a/docs/release/release-350.md
+++ b/docs/release/2021/release-350.md
diff --git a/docs/release/release-v3.3-rc3.md b/docs/release/2021/release-v3.3-rc3.md
index 1728eaa7..1728eaa7 100644
--- a/docs/release/release-v3.3-rc3.md
+++ b/docs/release/2021/release-v3.3-rc3.md
diff --git a/docs/release/release-351.md b/docs/release/2022/release-351.md
index eeeec747..eeeec747 100644
--- a/docs/release/release-351.md
+++ b/docs/release/2022/release-351.md
diff --git a/docs/release/release-352.md b/docs/release/2022/release-352.md
index 69566c32..69566c32 100644
--- a/docs/release/release-352.md
+++ b/docs/release/2022/release-352.md
diff --git a/docs/release/release-353.md b/docs/release/2022/release-353.md
index c5432f59..c5432f59 100644
--- a/docs/release/release-353.md
+++ b/docs/release/2022/release-353.md
diff --git a/docs/release/release-354.md b/docs/release/2022/release-354.md
index 58d4df31..58d4df31 100644
--- a/docs/release/release-354.md
+++ b/docs/release/2022/release-354.md
diff --git a/docs/release/release-355.md b/docs/release/2022/release-355.md
index ef32beee..ef32beee 100644
--- a/docs/release/release-355.md
+++ b/docs/release/2022/release-355.md
diff --git a/docs/release/release-356.md b/docs/release/2022/release-356.md
index 2ff94f99..2ff94f99 100644
--- a/docs/release/release-356.md
+++ b/docs/release/2022/release-356.md
diff --git a/docs/release/release-357.md b/docs/release/2022/release-357.md
index cad662c4..cad662c4 100644
--- a/docs/release/release-357.md
+++ b/docs/release/2022/release-357.md
diff --git a/docs/release/release-358.md b/docs/release/2022/release-358.md
index eba51cc9..eba51cc9 100644
--- a/docs/release/release-358.md
+++ b/docs/release/2022/release-358.md
diff --git a/docs/release/release-359.md b/docs/release/2022/release-359.md
index 583e5963..583e5963 100644
--- a/docs/release/release-359.md
+++ b/docs/release/2022/release-359.md
diff --git a/docs/release/release-360.md b/docs/release/2022/release-360.md
index a99bc12a..a99bc12a 100644
--- a/docs/release/release-360.md
+++ b/docs/release/2022/release-360.md
diff --git a/docs/release/release-361.md b/docs/release/2022/release-361.md
index 14bf2c70..14bf2c70 100644
--- a/docs/release/release-361.md
+++ b/docs/release/2022/release-361.md
diff --git a/docs/release/release-362.md b/docs/release/2022/release-362.md
index 6ba8f935..6ba8f935 100644
--- a/docs/release/release-362.md
+++ b/docs/release/2022/release-362.md
diff --git a/docs/release/release-363.md b/docs/release/2023/release-363.md
index ceb661c1..ceb661c1 100644
--- a/docs/release/release-363.md
+++ b/docs/release/2023/release-363.md
diff --git a/docs/release/release-364.md b/docs/release/2023/release-364.md
index 073d80cd..073d80cd 100644
--- a/docs/release/release-364.md
+++ b/docs/release/2023/release-364.md
diff --git a/docs/release/release-365.md b/docs/release/2023/release-365.md
index 82020af3..82020af3 100644
--- a/docs/release/release-365.md
+++ b/docs/release/2023/release-365.md
diff --git a/docs/release/release-366.md b/docs/release/2023/release-366.md
index 8928978a..8928978a 100644
--- a/docs/release/release-366.md
+++ b/docs/release/2023/release-366.md
diff --git a/docs/release/release-367.md b/docs/release/2023/release-367.md
index 22da5df0..22da5df0 100644
--- a/docs/release/release-367.md
+++ b/docs/release/2023/release-367.md
diff --git a/docs/release/release-368.md b/docs/release/2023/release-368.md
index f7a4c013..f7a4c013 100644
--- a/docs/release/release-368.md
+++ b/docs/release/2023/release-368.md
diff --git a/docs/release/release-369.md b/docs/release/2023/release-369.md
index c4bb35c5..c4bb35c5 100644
--- a/docs/release/release-369.md
+++ b/docs/release/2023/release-369.md
diff --git a/docs/release/release-370.md b/docs/release/2023/release-370.md
index 20aad732..20aad732 100644
--- a/docs/release/release-370.md
+++ b/docs/release/2023/release-370.md
diff --git a/docs/release/release-371.md b/docs/release/2023/release-371.md
index dd6efe5e..dd6efe5e 100644
--- a/docs/release/release-371.md
+++ b/docs/release/2023/release-371.md
diff --git a/docs/release/release-372.md b/docs/release/2023/release-372.md
index b8b034c0..b8b034c0 100644
--- a/docs/release/release-372.md
+++ b/docs/release/2023/release-372.md
diff --git a/docs/release/release-373.md b/docs/release/2023/release-373.md
index c88fc839..c88fc839 100644
--- a/docs/release/release-373.md
+++ b/docs/release/2023/release-373.md
diff --git a/docs/release/2023/release-374.md b/docs/release/2023/release-374.md
new file mode 100644
index 00000000..70400d72
--- /dev/null
+++ b/docs/release/2023/release-374.md
@@ -0,0 +1,45 @@
+Release 374 (TSG-23.12)
+* 新增自定义聚合函数：MEDIAN_HDR、QUANTILE_HDR、PERCENTILES_HDR、COUNT_DISTINCT、APPROX_COUNT_DISTINCT_HLLD、RATE(GAL-44、TSG-18048)
+* 新增自定义毫秒时间函数：FROM_UNIXTIME_MILLIS、UNIX_TIMESTAMP_MILLIS(GAL-449)
+* 新增HOS API, 支持文件上传、删除、文件元信息获取接口(TSG-17843、TSG-18698、TSG-18808)
+* 新增Monitor Schema，监测策略规则命中统计Schema(TSG-17851)
+* 新增Trace ID，日志中新增TraceID(同响应request_id)便于日志追踪(TSG-17851)
+* 删除过期Schema：radius_record、gtpc_record、radius_onff_log、sys_packet_capture_event、ctive_defence_event(TSG-17239、TSG-18119)
+* 删除ES组件应用及相关代码(GAL-439)
+* 重构Query API接口：管理SQL、DSL查询接口，支持批量获取(TSG-17269)
+* 重构Datasets API接口，初始化内置数据集(TSG-17787)
+* 重构Database API、Troubleshooting API(TSG-17843)
+* 重构Util API (TSG-17843)
+* 重构Knowledge Base API,对知识库文件管理接口进行重构(TSG-17843)
+* 修改liveChart，移除liveChart summary uniq_client_ip指标
+* 修改Dataset，field-discovery，filter增加 vsys_id过滤条件(TSG-18229)
+* 修改dataset，数monitor-event-timeline 只提供会话指标趋势(TSG-17952)
+* 修改配置，Hbase Saved Query结果表为tsg_galaxy:saved_query_result
+* 修改配置，HOS存储桶重命名为knowledge_base_bucket(GAL-453)
+* 修改Schema，适配DosEvent、ProxyEvent与AssessmentEvent结构变化(TSG-17819)
+* 修改Schema，基础字段app_path、protocol_path重命名(TSG-17828)
+* 修改Schema，修改文件路径补全对应HOS桶名
+* 修改Schema，更新fqdn_category_list字段dict_location数据字典引用位置
+* 修改Schema，，删除radius_match、gtpc_match 函数(TSG-17813)
+* 修改Schema，ssl_sni、dtls_sni、http_host、quic_sni字段新增sub_domain、get_value函数
+* 修改Schema，日志Schema部分字段补全Combine函数定义(TSG-17239)
+* 修改Schema，logicalType、constraints.type区分timestamp、unix_timestamp
+* 修改Schema，完善Log/Metric Schema 函数、Unit约束(TSG-18048)
+* 修改Schema，Proxy Event decoded_as 增加DoH协议字典
+* 修改Schema，更新流量Top类统计schema metric字典
+* 修改Schema，修改 metrics schema dimensions、metrics、filters自定义报告约束
+* 修改Schema，Monitor Events中未展示Monitor Mirrored相关字段(TSG-18602)
+* 修改Schema，Manipulation日志中应隐藏FQDN Category List、HTTP.Proxy Flags等字段(TSG-18577)
+* 修改Schema，修正traffic_shaping_rule_hits.doc中字段名(TSG-18639)
+* 扩展Schema，histogram支持MEDIAN_HDR函数
+* 扩展Schema，schema doc 增加time分类(TSG-17875)
+* 扩展Schema，新增公共字段ip_protocol，支持report统计(TSG-18203)
+* 扩展Schema，新增公共字段statistics_rule_list(TSG-18328)
+* 扩展Schema，VoIP Record decoded_as 增加VoIP字典值(TSG-18445)
+* 扩展Schema，日志Schema支持表达式字段(TSG-18511)
+* 扩展Schema，object_statistics 新增object_type字段及字典数据(TSG-17960)
+* 扩展Schema，描述可作为维度的时间字段(TSG-18662)
+* 优化样例数据接口，优化Chart预览样例数据的生成方式(TSG-18645、TSG-18673)
+
+
+
diff --git a/docs/release/release-374.md b/docs/release/release-374.md
deleted file mode 100644
index 723ff0d3..00000000
--- a/docs/release/release-374.md
+++ /dev/null
@@ -1,2 +0,0 @@
-Release 374 (TSG-23.12)
-* 
\ No newline at end of file
diff --git a/docs/release/release-375.md b/docs/release/release-375.md
new file mode 100644
index 00000000..4e4dbd1f
--- /dev/null
+++ b/docs/release/release-375.md
@@ -0,0 +1,12 @@
+Release 375 (TSG-24.02)
+* 修改Schema，修改部分Metric聚合及延迟粒度(TSG-18857)
+* 修改Schema，Security Rule Hits Action数据字典去除 Monitor(TSG-18885)
+* 修改Schema，字段tunnels重命名为encapsulation(TSG-19249)
+* 修正字段发现统计结果查询失败(TSG-19354)
+* 修改Schema，解决Encapsulation内容重复显示的问题(TSG-19347)
+* 修改Schema，移除format function定义(GAL-488)
+* 完善Schema校验接口(TSG-18390)
+* 优化TIME_FLOOR_WITH_FILL函数转义(TSG-19375)
+* 修改Dockerfile，更新默认路径为/opt/galaxy-qgw-service(GAL-502)
+* 修改Schema，修正Schema部分字段数据类型(TSG-19671)
+* 修改Schema，aggregated_metric_unit补充dns_response_latency_ms，ssl_handshake_latency_ms字段(TSG-19592)
\ No newline at end of file
diff --git a/docs/release/release-376.md b/docs/release/release-376.md
new file mode 100644
index 00000000..55215bdc
--- /dev/null
+++ b/docs/release/release-376.md
@@ -0,0 +1,11 @@
+Release 376 (TSG-24.03)
+* 隐藏Top Metrics Schema metric列，Dataset不再按照Metric定义不同数据源(TSG-19563)
+* 优化Dry Run查询的样例数据生成方式(TSG-19467)
+* 完善子查询优化判断逻辑，补充自定义聚合(TSG-19921)
+* 修复样例数据，修复结果维度列数据重复问题(TSG-19890)
+* 修复样例数据，修复别名存在特殊字符导致JSONPath解析失败问题(TSG-19870)
+* 修复样例数据，修复时序数据多维度情况下数据不完整问题(TSG-19892)
+* 扩展Schema，在tunnel_type 描述GRE(TSG-19518)
+* 扩展Schema，扩展表达式字段：internal_port_list, external_port_list, interal_asn_list, external_asn_list(TSG-19579)
+* 完善查询ClickHouse，以适配时区差异(GAL-520)
+* 完善任务提交，针对Saved Query Job不做判重逻辑(TSG-19990)
\ No newline at end of file
diff --git a/docs/release/release-377.md b/docs/release/release-377.md
new file mode 100644
index 00000000..f1e905b6
--- /dev/null
+++ b/docs/release/release-377.md
@@ -0,0 +1,29 @@
+Release 377 (TSG-24.04)
+* 新增Schema，新增Dos Protection Rule Hits Schema 和 dataset(TSG-19777)
+* 新增Schema，新增DoS Protection Rule Metric Schema(TSG-19845)
+* 扩展Schema，Statistics Rule Metric Schema 和 dataset 增加client_asn 和 server_asn(TSG-19744)
+* 扩展Schema，Statistics Rule Metric Schema 和 dataset 增加client_country 和 server_countryTSG-19886)
+* 扩展Schema，增加公共字段tunnel_id_list(TSG-20333)
+* 扩展Schema，TopK Schema 维度增加rank_by(TSG-20020)
+* 扩展Schema，新增基于Client/Server Geolocation公共字段(TSG-20205)
+* 扩展RATE函数，支持不同时间粒度(TSG-19773)
+* 修改Query API，exec_mode 修改为 execution_mode、formatType修改为output_mode(TSG-19957)
+* 修改Query API，完善字段Meta类型与结果值类型不一致问题、Meta移除Category(TSG-19957)
+* 修改Query API，规范request_id、job_id 为标准UUID(TSG-19957)
+* 修改Dataset，时间范围变量统一命名为start_time，end_time(TSG-19957)
+* 修改Schema，TopK Schema及Dataset适配修改(TSG-20020)
+* 修改Schema，DoS Event 增加DoS Protection Rule相关字段(TSG-20103)
+* 修改Schema，修改dup_traffic_flag、ssl_esni_flag、ssl_ech_flag数据字典(TSG-20198)
+* 修改Schema，Proxy Manipulation日志隐藏server_os_desc、client_os_desc字段(TSG-20210)
+* 删除Schema，删除Top Subscriber Schema及Dataset(TSG-20020)
+* 删除过期代码，删除非结构化数据文件接口、实时数据分析接口、liveChart冗余代码(TSG-20173)
+* 优化pom依赖，删除有冲突的无效依赖(GAL-512)
+* 优化RATE函数，结果保留两位小数(TSG-20131)
+* 优化Dataset，使用RATE函数替代SubQuery 嵌套(TSG-19957)
+* 扩展Schema，完善Dos Events attack_type数据字典(TSG-20551)
+* 扩展Schema，Schema新增mail_starttls_flag字段(TSG-20751)
+* 修改Schema，address_type 数据字典调整(TSG-20589)
+* 修改Database API，解决公共Schema更新未即时生效的问题(GAL-540)
+* 修改Dataset，修正proxy-manipulation-policy-rule-hits-by-action 数据集(TSG-20544)
+* 修改Schema，Proxy Intercept/Manipulation 隐藏 tunnel_id_list、tunnel_endpoint_a_desc、tunnel_endpoint_b_desc字段(TSG-20675)
+* 优化储存配额接口，优化Get Storage quota 接口，druid、xxl-job数据不一致导致数据为空问题
diff --git a/docs/release/release-378.md b/docs/release/release-378.md
new file mode 100644
index 00000000..a1813d2f
--- /dev/null
+++ b/docs/release/release-378.md
@@ -0,0 +1,14 @@
+Release 378 (TSG-24.05)
+* 新增Schema, 新增路径数据遥测日志datapath_telemetry_record(TSG-20473)
+* 新增Dataset, 新增数据路径遥测相关业务数据集: count、list、packet combine(TSG-20473)
+* 新增Pcapng合并功能, 扩展DSL接口支持Packet Combine合并功能(TSG-20473)
+* 扩展Schema, 新增app_extra_info公共字段(TSG-20783)
+* 扩展Schema, Statistics Rule 新增 c2s/s2c_ttl、c2s/s2c_link_id 字段，支持Statistics模版维度(TSG-20725)
+* 扩展Schema, 日志新增公共字段c2s_ttl、s2c_ttl(TSG-20811)
+* 扩展Schema, Link ID支持 Report Filter
+* 扩展Schema, 新增公共字段direction(TSG-20925)
+* 扩展操作符函数, 支持IS NULL、IS NOT NULL检索(TSG-20925) 
+* 修改Schema, VoIP Record删除与SIP/RTP协议无关字段(TSG-20765)
+* 修改Schema, 设置表达式_rule_id 字段不支持日志检索
+* 修改HOS API, 参数命名遵循下划线规范(TSG-20730)
+* 修改schema, 安全事件日志隐藏monitor_mirrored_pkts，monitor_mirrored_bytes字段
diff --git a/docs/release/release-379.md b/docs/release/release-379.md
new file mode 100644
index 00000000..952ebfb1
--- /dev/null
+++ b/docs/release/release-379.md
@@ -0,0 +1,12 @@
+Release 379 (TSG-24.06)
+* 扩展Schema, 新增公共字段app_category(TSG-21392)
+* 新增Schema, 新增流量透视Metric traffic_sketch_metric(TSG-20989)
+* 新增Dataset, 新增Traffic Spectrum Dataset(TSG-20989)
+* 扩展DSL接口, 支持Traffic Spectrum相关功能(TSG-20989)
+* 修正Schema, 修正Packets 相关字段统计单位(TSG-21471)
+* 调整HDR_GET_PERCENTILES percentileTicksPerHalfDistance参数默认值为5 (TSG-21466)
+* 适配通过nginx查询clickhouse(GAL-585)
+* 优化TopN优化最内层SQL，Limit下限制设置为1000(TSG-21485)
+* 优化Drilldown查询，结果数据标识Subtotal rows(TSG-21477)
+* 完善字段发现，修正Array类型字段发现未统计空记录(TSG-21480)
+* 修改Schema，flags Asymmetric 标注已过期 Deprecated
\ No newline at end of file
diff --git a/docs/release/release-380.md b/docs/release/release-380.md
new file mode 100644
index 00000000..b12c499d
--- /dev/null
+++ b/docs/release/release-380.md
@@ -0,0 +1,13 @@
+Release 380 (TSG-24.07)
+* 新增储存配额 sys_storage_event表(TSG-21555)
+* 扩展Schema，IP类型字段支持 isIPAddressInRange(TSG-21544)
+* 扩展Schema，Statistics Rule Metric 增加IP Fragments字段(TSG-21637)
+* 扩展Schema，数据路径遥测日志新增字段egress_action(TSG-21768)
+* 扩展时间粒度函数，DSL、SQL Query Job支持默认时间粒度(TSG-21420)
+* 扩展Function，转义isIPAddressInRange，解决空值导致函数执行异常问题
+* 扩展Traffic Spectrum 二部图，支持时间分片查询
+* 修正Traffic Spectrum Throughput 流量曲线粒度(TSG-21648)
+* 完善样例数据，避免schema dict值数量小于随机结果数量时，生成数据异常问题
+* 完善审计日志，查询网关具备记录高延迟查询 SQL 的功能(GAL-603)
+* 优化储存配额接口，新增max_usage、基于mariadb获取数据、删除冗余代码(TSG-21554)
+* 优化Schema，统一命名:Client-to-Server、Server-to-Client(TSG-21651)
\ No newline at end of file