05-Kapacitor监控案例-B-降采样

• 降采样
  • 概念
  • Batch Task 要求
  • 解题过程
    • 1. 画出DAG图
    • 2. 编写 TICKscript
    • 3. 启动后查看日志
    • 4. 查看数据库

降采样

概念

维基百科解释

在数位信号处理领域中，降采样，又作减采集,是一种多速率数字信号处理的技术或是降低信号采样率的过程，通常用于降低数据传输速率或者数据大小。

在具体的业务场景当中，我们也往往不需要太高的精度，比如查看一年的股票走势，我们把精度下降到天为单位就能满足需求，同时也能提高处理速度。

Batch Task 要求

批量获取 Influxdb数据库中

• 库名database：telegraf.autogen
• 测量名measurement：cpu
• 时间窗口:近1天
• 计算每个主机的cpu=cpu-total的usage_idle每小时的平均值

输出到日志中；

输出到influxdb数据中：

• 库：output_db.autogen
• 测量名:cpu_one_hour

解题过程

> SELECT mean("usage_idle") as mean_usage_idle FROM "cpu" WHERE cpu = 'cpu-total' and time >= now()-1d  GROUP BY time(1h),cpu,host fill(null)
name: cpu
tags: cpu=cpu-total, host=influxdb
time                mean_usage_idle
----                ---------------
1567918800000000000 97.93619008178243
1567922400000000000 97.93562487721887
1567926000000000000 97.94969924086021
1567929600000000000 97.94223179589362
1567933200000000000 97.91641772738491
1567936800000000000 97.9314543523225
1567940400000000000 97.94060559301319
1567944000000000000 97.90147239022043
1567947600000000000 97.76035767048423
1567951200000000000 97.86604814199909
1567954800000000000 97.93333384474244
1567958400000000000 97.92344141186854
1567962000000000000 97.96961809140475
1567965600000000000 97.98192006803991
1567969200000000000 97.96356564256739
1567972800000000000 98.01916963501021
1567976400000000000 97.99284401689296
1567980000000000000 97.99148788645532
1567983600000000000 97.97212285028817
1567987200000000000 98.00985809717609
1567990800000000000 97.9692064309443
1567994400000000000 97.94306224519104
1567998000000000000 97.95517428364671
1568001600000000000 97.92946756545597
1568005200000000000 97.92342386210187

1. 画出DAG图

graph LR;
  A(BatchNode)-->B
  B(QueryNode)-->C
  C(LogNode)-->F(InfluxDBOutNode)

2. 编写 TICKscript

// cpu_batch.tick
dbrp "telegraf"."autogen"

batch
    |query('''SELECT mean("usage_idle") as mean_usage_idle FROM telegraf.autogen.cpu WHERE cpu = 'cpu-total'
         ''')
      .period(24h)
      .every(24h)
      .groupBy(time(1h), 'cpu', 'host')
    |log()
    |influxDBOut()
        .create()
        .database('output_db')
        .retentionPolicy('autogen')
        .measurement('cpu_one_hour')

每天查询一次InfluxDB; 返回的时间窗口跨越24小时，将数据点从1分钟降低到1h一个点。

3. 启动后查看日志

ts=2019-09-09T14:38:53.829+08:00 lvl=debug msg="starting next batch query" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 query="SELECT mean(usage_idle) AS mean_usage_idle FROM telegraf.autogen.cpu WHERE cpu = 'cpu-total' AND time >= '2019-09-08T06:38:53.829083954Z' AND time < '2019-09-09T06:38:53.829083954Z' GROUP BY time(1h, 0s), cpu, host"
ts=2019-09-09T14:38:53.831+08:00 lvl=info msg="begin batch" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb time=2019-09-09T06:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.9417652419907 time=2019-09-08T06:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.94969924086021 time=2019-09-08T07:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.94223179589362 time=2019-09-08T08:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.91641772738491 time=2019-09-08T09:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.9314543523225 time=2019-09-08T10:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.94060559301319 time=2019-09-08T11:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.90147239022043 time=2019-09-08T12:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_host=influxdb tag_cpu=cpu-total field_mean_usage_idle=97.76035767048423 time=2019-09-08T13:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_host=influxdb tag_cpu=cpu-total field_mean_usage_idle=97.86604814199909 time=2019-09-08T14:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.93333384474244 time=2019-09-08T15:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_host=influxdb tag_cpu=cpu-total field_mean_usage_idle=97.92344141186854 time=2019-09-08T16:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.96961809140475 time=2019-09-08T17:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.98192006803991 time=2019-09-08T18:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_host=influxdb tag_cpu=cpu-total field_mean_usage_idle=97.96356564256739 time=2019-09-08T19:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=98.01916963501021 time=2019-09-08T20:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.99284401689296 time=2019-09-08T21:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.99148788645532 time=2019-09-08T22:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_host=influxdb tag_cpu=cpu-total field_mean_usage_idle=97.97212285028817 time=2019-09-08T23:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=98.00985809717609 time=2019-09-09T00:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.9692064309443 time=2019-09-09T01:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_host=influxdb tag_cpu=cpu-total field_mean_usage_idle=97.94306224519104 time=2019-09-09T02:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.95517428364671 time=2019-09-09T03:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.92946756545597 time=2019-09-09T04:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.92929794506509 time=2019-09-09T05:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="batch point" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb field_mean_usage_idle=97.95361354221568 time=2019-09-09T06:00:00Z
ts=2019-09-09T14:38:53.832+08:00 lvl=info msg="end batch" service=kapacitor task_master=main task=cpu_batch node=influxdb_out3 prefix= name=cpu group=cpu=cpu-total,host=influxdb tag_cpu=cpu-total tag_host=influxdb time=2019-09-09T06:00:00Z

4. 查看数据库

[root@tick:/var/lib/kapacitor/task]# zy_influx -database output_db -execute 'select * from cpu_one_hour'
name: cpu_one_hour
time                cpu       host     mean_usage_idle
----                ---       ----     ---------------
1567922400000000000 cpu-total influxdb 97.93541733664286
1567926000000000000 cpu-total influxdb 97.94969924086021
1567929600000000000 cpu-total influxdb 97.94223179589362
1567933200000000000 cpu-total influxdb 97.91641772738491
1567936800000000000 cpu-total influxdb 97.9314543523225
1567940400000000000 cpu-total influxdb 97.94060559301319
1567944000000000000 cpu-total influxdb 97.90147239022043
1567947600000000000 cpu-total influxdb 97.76035767048423
1567951200000000000 cpu-total influxdb 97.86604814199909
1567954800000000000 cpu-total influxdb 97.93333384474244
1567958400000000000 cpu-total influxdb 97.92344141186854
1567962000000000000 cpu-total influxdb 97.96961809140475
1567965600000000000 cpu-total influxdb 97.98192006803991
1567969200000000000 cpu-total influxdb 97.96356564256739
1567972800000000000 cpu-total influxdb 98.01916963501021
1567976400000000000 cpu-total influxdb 97.99284401689296
1567980000000000000 cpu-total influxdb 97.99148788645532
1567983600000000000 cpu-total influxdb 97.97212285028817
1567987200000000000 cpu-total influxdb 98.00985809717609
1567990800000000000 cpu-total influxdb 97.9692064309443
1567994400000000000 cpu-total influxdb 97.94306224519104
1567998000000000000 cpu-total influxdb 97.95517428364671
1568001600000000000 cpu-total influxdb 97.92946756545597
1568005200000000000 cpu-total influxdb 97.92929794506509
1568008800000000000 cpu-total influxdb 97.95526618492178