diff options
| -rw-r--r-- | readme.md | 415 |
1 files changed, 242 insertions, 173 deletions
@@ -1,199 +1,268 @@ -# Field Stat 2简介 - -Field Stat是一个运行状态输出与统计库。 - -## 本地显示风格(style) - - 本地显示会输出到屏幕或文件，可选择输出瞬时值 `FS_CALC_SPEED`或累计值`FS_CALC_CURRENT`。 - - 允许在`FS_start`启动后注册非Column的统计域。 - -### 域（Field ） - - `FS_STYLE_FIELD` 同时输出累计值(sum)和瞬时值(speed/s，1秒内的变化值)。 - -``` +# Fieldstat3 +Fieldstat3 is a running state output and statistics library. Fieldtat3 inherits the functions of Fieldstat2. +## Metric ype +The metric type of Fieldstat is similar to promethues. As follows. +- Counter +- Gauge +- Histogram +- Summary + +## Exporter Type +Fieldstat3 has three exporter types. As follows. +| Exporter Type | Decription | +| --- | --- | +| Local | Export metric to a specified local file in a custom format. | +| Prometheus | Export metric by prometheus endpoint in the prometheus Format. | +| Line Protocol | Export metric to Line Protocol Server(i.e., telegraf) in Line Portocol Format. | + +### Local +#### Counter Metric Type +The Local exporter exports the changing(speed/s) and current(sum) value of the Counter metrics. As follows. +```txt field_00 field_01 sum 0 184300 speed/s 0 92150 ``` - - 函数接口 - -```c -field_ids[i]=FS_register(handle, FS_STYLE_FIELD, FS_CALC_CURRENT, "field_01"); -``` - -### 状态（Status） - - `FS_STYLE_STATUS` ，可以输出当前值（FS_CALC_CURRENT）或速度值（FS_CALC_SPEED）。 - -```c +#### Gauge Metric Type +The local exporter exports the current value of the Gauge metrics. as follows. +```txt status_00: 100 status_01: 10 ``` - - 输出前值（FS_CALC_CURRENT）时，StatsD输出会采用Histogram类型，以便telegraf对其进行聚合。例如下面的grafana查询语句： - -`SELECT sum("value_sum") / 5 FROM "[tango_cache]SESSION_REDIS" WHERE $timeFilter GROUP BY time(10s) fill(linear)` - - 该语句在聚合多个节点上SESSION_REDIS的总数，group by 的时间为10s，因为FieldStat的输出间隔为2秒（初始化时由`STAT_CYCLE`参数设置，默认2s），所以sum("value_sum") 的值要再除以5。 - - 注意：在使用多个FieldStat实例时，`STAT_CYCLE`必须设定为相同的值。 - -函数接口 - +#### Summary Metric Type +The local exporter exports the Summary metrics in a table format. As follows. ``` -status_ids[i]=FS_register(handle, FS_STYLE_STATUS, FS_CALC_CURRENT, "status_01"); +summary 50% 80% 95% MAX MIN AVG STDDEV CNT +ssl_up(ms) 0 0 0 0 0 0.00 0.00 0 +peek_sni(ms) 0 0 0 0 0 0.00 0.00 0 +ssl_down(ms) 0 0 0 0 0 0.00 0.00 0 +ask_kyr(ms) 0 0 0 0 0 0.00 0.00 0 ``` - -### 表格 （Line&Column） - - `FS_STYLE_LINE`和`FS_STYLE_COLUMN `输出当前值或速度值。 - +#### Histogram Metric Type +The local exporter exports the Histogram metrics in a table format. As follows. ``` -___________________________________________________________________________________________________ - column_0 column_1 column_3 column_4 column_5 c0/c1 -line_0 2.76e+04 5.53e+04 1.11e+05 1.38e+05 1.66e+05 5.00e-01 -line_1 2.76e+04 5.53e+04 1.11e+05 1.38e+05 1.66e+05 5.00e-01 -line_2 2.76e+04 5.53e+04 1.11e+05 1.38e+05 1.66e+05 5.00e-01 -line_3 2.76e+04 5.53e+04 1.11e+05 1.38e+05 1.66e+05 5.00e-01 -line_4 2.76e+04 5.53e+04 1.11e+05 1.38e+05 1.66e+05 5.00e-01 -line_5 2.76e+04 5.53e+04 1.11e+05 1.38e+05 1.66e+05 5.00e-01 -line_6 2.76e+04 5.53e+04 1.11e+05 1.38e+05 1.66e+05 5.00e-01 +histogram le=50 le=80 le=95 MAX MIN AVG STDDEV CNT +ssl_up(ms) 0 0 0 0 0 0.00 0.00 0 +peek_sni(ms) 0 0 0 0 0 0.00 0.00 0 +ssl_down(ms) 0 0 0 0 0 0.00 0.00 0 +ask_kyr(ms) 0 0 0 0 0 0.00 0.00 0 ``` - - 函数接口 API - +### Prometheus +#### Counter Metric Type +The prometheus exporter exports the current value of the Counter metrics. As follows. +```txt +counter_00{app_name="counter_app_name"} 1\n ``` -line_ids[i]=FS_register(handle, FS_STYLE_LINE, FS_CALC_CURRENT, "line_0"); -column_ids[i]=FS_register(handle, FS_STYLE_COLUMN, FS_CALC_SPEED,"column_0"); +#### Gauge Metric Type +The prometheus exporter exports the current value of the Gauge metrics. as follows. +```txt +gauge_00{app_name="gauge_app_name"} 1\n ``` - - - -### 分布图 (Histogram) - - `FS_STYLE_HISTOGRAM` 输出事件的数值分布，如延迟，文件长度等。 - +#### Summary Metric Type +The prometheus exporter exports the Summary metrics. As follows. ``` -histogram 50% 80% 95% MAX MIN AVG STDDEV CNT -ssl_up(ms) 0 0 0 0 0 0.00 0.00 0 -peek_sni(ms) 0 0 0 0 0 0.00 0.00 0 -ssl_down(ms) 0 0 0 0 0 0.00 0.00 0 -ask_kyr(ms) 0 0 0 0 0 0.00 0.00 0 +prometheus_00{app_name="prometheus_app_name",quantile="0.50"} 50\n +prometheus_00{app_name="prometheus_app_name",quantile="0.10"} 10\n +prometheus_00{app_name="prometheus_app_name",quantile="0.80"} 80\n +prometheus_00{app_name="prometheus_app_name",quantile="0.90"} 90\n +prometheus_00{app_name="prometheus_app_name",quantile="0.95"} 95\n +prometheus_00{app_name="prometheus_app_name",quantile="0.99"} 99\n +prometheus_00_count{app_name="prometheus_app_name"} 100\n" +prometheus_00_sum{app_name="prometheus_app_name"} 5050\n" ``` - - 默认输出 50%，80.00%，90%，95%和99%的比例， 可以通过 `FS_set_para` 函数的`HISTOGRAM_GLOBAL_BINS`选项修改。 - - 缩写定义： - -- MAX 最大值 - -- MIN 最小值 - -- AVG 平均值 - -- STDDEV 标准差 - -- CNT 计数 - - 函数接口 - +#### Histogram Metric Type +The local exporter exports the Histogram metrics in a table format. As follows. ``` -fs_metric_sql_exec=FS_register_histogram(g_info.fs_handle, //Field Stat句柄 - FS_CALC_CURRENT, //输出累计值或瞬时值 - "sql_exec(us)", //统计项名称，字符串 - 1, //可追踪的最小值 - 1000000, //可追踪的最大值 - 2); //精度，即小数点后几位，范围1~4 +histogram_00_bucket{app_name="histogram_00_app_name",le="10.00"} 10\n +histogram_00_bucket{app_name="histogram_00_app_name",le="50.00"} 50\n +histogram_00_bucket{app_name="histogram_00_app_name",le="90.00"} 90\n +histogram_00_bucket{app_name="histogram_00_app_name",le="95.00"} 95\n +histogram_00_bucket{app_name="histogram_00_app_name",le="99.00"} 99\n +histogram_00_bucket{app_name="histogram_00_app_name",le="80.00"} 80\n +histogram_00_count{app_name="histogram_00_app_name"} 100\n +histogram_00_sum{app_name="histogram_00_app_name"} 5050\n ``` - - - -## 远程输出 - - 可以输出为[StatsD](https://github.com/influxdata/telegraf/tree/master/plugins/inputs/statsd)格式或 [InfluxDB Inline Protocol](https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_tutorial/)格式，通过UDP协议发送给聚合器，如telegraf。 聚合器接收并聚合后，写入时序数据库，如InfluxDB，然后通过图形界面展示，如Grafana。 - -StatsD 输出counter和histogram两种metric，metric name命名遵循 - -- Field和Status： ` [APP_NAME]field_name:value` -- Column和Line：`[APP_NAME]line#column:value` - -Influx Line命名规则为 - +### Line Protocol +#### Counter Metric Type +The Line Protocol exporter exports the current value of the Counter metrics. As follows. +```txt +counter_00,app_name=counter_00_app_name counter_00=0\n ``` -measurement, tag, key1=value1, key2=value2 -line，app_name=APPNAME column1=value,column2=value +#### Gauge Metric Type +The Line Protocol exporter exports the current value of the Gauge metrics. as follows. +```txt +gauge_00,app_name=gauge_00_app_name gauge_00=0\n ``` - - 可以通过 `FS_set_para`函数 的`STATS_SERVER_IP ` 、`STATS_SERVER_PORT`选项，设置服务器的IP和端口。通过`STATS_FORMAT`选项设置输出格式为`FS_OUTPUT_STATSD`或`FS_OUTPUT_INFLUX_LINE`。 - - 服务端搭建流程参见[《【StatsD监控】基于Telegraf+InfluxDB+grafana展示（非Docker方案）》](https://blog.csdn.net/littlefang/article/details/78473161)。聚合Histogram建议修改telegraf.conf的以下配置项： - +#### Summary Metric Type +The Line Protocol exporter exports the Summary metrics. As follows. ``` -[[inputs.statsd]] - protocol = "udp4" - service_address = ":8125" - percentiles = [20,30,40,50,60,70,80,90,95,99] - percentile_limit = 100000000 +summary_00,app_name=summary_00_app_name P50=0,P80=0,P99=0,max=0,min=0,avg=0,stddev=0.00,cnt=0\n ``` - - - -## 统计 - -### 记录统计值 - - 通过`FS_operate`函数操作统计域，函数内部使用原子变量，无锁且线程安全。 - +#### Histogram Metric Type +The Line Protocol exporter exports the Histogram metrics in a table format. As follows. ``` -//handle: FieldStat句柄 -//id: FS_register_**返回的ID -//column_id: 列ID，非column类型填0 -//filed_op: FS_OP_ADD: + value -// FS_OP_SUB: - value -// FS_OP_SET: = value -int FS_operate(screen_stat_handle_t handle,int id,int column_id,enum field_op op,long long value); +summary_00,app_name=summary_00_app_name le50=0,le80=0,le99=0,max=0,min=0,avg=0,stddev=0.00,cnt=0\n ``` - -### 记录事件耗时 - - 应使用clock_gettime函数获得事件的耗时，[永远不要用gettimeofday计算时间](https://blog.habets.se/2010/09/gettimeofday-should-never-be-used-to-measure-time.html)。 - - 也可使用field_stat2.h中提供的简单封装。 - +## Table format +The table format is a separated concept. It is a reencapsulation of metrics whose metric type is counter and gauge. +### Local Exporter +The Local Exporter exports the table in the following manner. ``` -struct timespec start; -record_time_start(&start); -//do something -long eplapses=record_time_elapse_us(&start); -FS_operate(fs_handle, histogram_id, 0, FS_OP_SET, eplapsed); +__________________________________ + column_0 column_1 +line_0 0 1 +line_1 1 2 ``` +### Prometheus Exporter +The Prometheus Exporter exports the table in the following manner. +``` +column_0{app_name="table_0_app_name",table_name="table_0",line_name="line_0"} 0\n +column_1{app_name="table_0_app_name",table_name="table_0",line_name="line_0"} 1\n +column_0{app_name="table_0_app_name",table_name="table_0",line_name="line_1"} 1\n +column_1{app_name="table_0_app_name",table_name="table_0",line_name="line_1"} 2\n +``` +### Line Protocol Exporter +The Line Protocol Exporter exports the table in the following manner. +``` +line_0,app_name=table_0_app_name,table_name=table_0 column_0=0,column_1=1\n +line_1,app_name=table_0_app_name,table_name=table_0 column_0=1,column_1=2\n +``` +## Static and Dynamic Interface +The Fieldstat3 has static and dynamic interfaces. +### Static Interface +The static interfaces provide prometheus exporter, local exporter, and Line protocol expoeter. +There are used in environments with low performance requirements. +### Dynamic Interface +The static interfaces only provide Line Protocol expoeter. +There are used in environments with high performance requirements. + +## Usage +Here are some examples of calling function interfaces. +### Using Statics Interface +#### Create Fieldstat3 Instance +```c +struct fieldstat_instance *instance = fieldstat_instance_new("fs3_example"); +``` +#### Enable Line Protocol Exporter +```c +int ret = fieldstat_set_line_protocol_server(instance, "127.0.0.1", 8001); +``` +#### Enable Prometheus Exporter +```c +int ret = fieldstat_global_enable_prometheus_endpoint(9020, "/metrcis"); +int ret = fieldstat_enable_prometheus_output(instance); +``` +#### Enable Local Exporter +```c +int ret = fieldstat_set_local_output(instance, "/tmp/fieldstat3.txt", "default"); +``` +#### Register and Operate Table +```c +int ret = -1; +const char *columns[] = {"column0", "column1", "column2", "column3"}; +enum field_type columns_type[] = {FIELD_TYPE_COUNTER,FIELD_TYPE_COUNTER, FIELD_TYPE_COUNTER, FIELD_TYPE_COUNTER}; +int table_id = fieldstat_register_table(instance, "table_name", columns, columns_type, sizeof(columns)/sizeof(column[0])); + +int output_metrics_id[4]; +int ret = fieldstat_register_table_row(instance, table_id, "row0", NULL, 0, output_metrics_id); +//Incrby +ret = fieldstat_value_incrby(instance, output_metric_ids[0], 1); +//Decrby +ret = fieldstat_value_decrby(instance, output_metric_ids[0], 1); +//Set +ret = fieldstat_value_set(instance, output_metric_ids[0], 1); +``` +#### Register and Operate Counter +```c +int ret = 0; +int mid = fieldstat_register(instance, FIELD_TYPE_COUNTER, "counter00", NULL, 0); +//Incrby +ret = fieldstat_value_incrby(instance, mid, 10); +//Decrby +ret = fieldstat_value_decrby(instance, mid, 10); +//Set +ret = fieldstat_value_set(instance, mid, 10); +``` +#### Register and Operate Gague +```c +int ret = 0; +int mid = fieldstat_register(instance, FIELD_TYPE_GAUGE, "gauge00", NULL, 0); +//Incrby +ret = fieldstat_value_incrby(instance, mid, 10); +//Decrby +ret = fieldstat_value_decrby(instance, mid, 10); +//Set +ret = fieldstat_value_set(instance, mid, 10); +``` +#### Register and Operate Histogram +```c +const char * bin = "10,20,30,40,50,60,70,80,90"; +int mid = fieldstat_register_histogram(instance, "htr", NULL, 0, bins, 1, 1000, 2, 0); +//Append value into histogram set. +ret = fieldstat_value_set(instance, mid, 10); +``` +#### Register and Operate Summary +```c +const char * bins = "0.1,0.2,0.3,0.4,0.5,0.8,0.9,0.95,0.99"; +int mid = fieldstat_register_summary(instance, "htr", NULL, 0, bins, 1, 1000, 2, 0); +//Append value into summary set. +ret = fieldstat_value_set(instance, mid, 10); +``` +#### Start Fieldstat Instance +```c +fieldstat_instance_start(instance); +``` - -## 使用建议 - -### 命名建议 - - 在命名时，遵循按照以下规则： - -1. 带上单位，如bytes或kb，微秒us或毫秒ms等。 -2. 除若干单词首字母组成的缩写外，不要大写。如Kernel Network Interface 可以缩写为KNI，packet缩写为pkt，而不是PKT -3. 一般不要超过10个字节，不然本地刷新不美观。 - - htable->htab, read->rd，write->wr，send ->tx，receive->rx - - - send packets，可缩写为tx_pkts，send bytes可缩写为tx_bytes - - 禁止注册名称包含 `|:\n\r. \t<>[]#!@`的统计域。 - -### 显示字段数量 - - Status和Field每行固定展示8个统计值。Column和Histogram取决于注册数量，建议Column一般不超过8个，Histogram的bins不超过5个。 - - 通过`FS_set_para`函数的 `ID_INVISBLE`不显示的字段，可以节约屏幕空间；通过 `FS_register_ratio`计算两个统计字段的比值。 - -### 本地结果查看 - - 在shell执行`watch -d cat 日志文件`，观察实时的输出结果。 - +### Using Dynamic Interface +#### Create Fieldstat3 Instance +```c +int n_threads = 0; +struct fieldstat_dynamic_instance *instance = fieldstat_dynamic_instance_new("fs3_example", n_threads); +``` +#### Enable Line Protocol Exporter +```c +int ret = fieldstat_dynamic_set_line_protocol_server(instance, "127.0.0.1", 8001); +``` +#### Register and Operate Table +```c +int ret = -1; +int thread_id = 0; +int out_column_ids[4]; + +const char *columns[] = {"column0", "column1", "column2", "column3"}; +enum field_type columns_type[] = {FIELD_TYPE_COUNTER,FIELD_TYPE_COUNTER, FIELD_TYPE_COUNTER, FIELD_TYPE_COUNTER}; +int table_id = fieldstat_register_dynamic_table(instance, "table_name", columns, columns_type, sizeof(columns)/sizeof(columns[0]), out_column_ids); +//Incrby +ret = fieldstat_dynamic_table_metric_value_incrby(instance, table_id, out_column_ids[0], "table_name", 1, NULL, 0, thread_id); +//Decrby +ret = fieldstat_dynamic_table_metric_value_decrby(instance, table_id, out_column_ids[0], "table_name", 1, NULL, 0, thread_id); +//Set +ret = fieldstat_dynamic_table_metric_value_set(instance, table_id, out_column_ids[0], "table_name", 1, NULL, 0, thread_id); +``` +#### Register and Operate Counter +```c +int ret = 0; +int thread_id = 0; +//Incrby +ret = fieldstat_dynamic_metric_value_incrby(instance, FIELD_TYPE_COUNTER, "counter00", 1, NULL, 0, thread_id); +//Decrby +ret = fieldstat_dynamic_metric_value_decrby(instance, FIELD_TYPE_COUNTER, "counter00", 1, NULL, 0, thread_id); +//Set +ret = fieldstat_dynamic_metric_value_set(instance, FIELD_TYPE_COUNTER, "counter00", 1, NULL, 0, thread_id); +``` +#### Register and Operate Gauge +```c +int ret = 0; +int thread_id = 0; +//Incrby +ret = fieldstat_dynamic_metric_value_incrby(instance, FIELD_TYPE_GAUGE, "counter00", 1, NULL, 0, thread_id); +//Decrby +ret = fieldstat_dynamic_metric_value_decrby(instance, FIELD_TYPE_GAUGE, "counter00", 1, NULL, 0, thread_id); +//Set +ret = fieldstat_dynamic_metric_value_set(instance, FIELD_TYPE_GAUGE, "counter00", 1, NULL, 0, thread_id); +``` + +#### Start Fieldstat Instance +```c +fieldstat_dynamic_instance_start(instance); +``` |