machine_memory_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="node01",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="node01",kubernetes_io_os="linux"} 4092948480
Но в байтах ненаглядно, поэтому воспользуемся PromQL для перевода в Gb: machine_memory_bytes / 1000 / 1000 / 1000
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="controlplane",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="controlplane",kubernetes_io_os="linux"} 2.096992256
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="node01",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="node01",kubernetes_io_os="linux"} 4.09294848
Введём для memory_bytes для поиска container_memory_usage_bytes использованной памяти. Список содержит все контейнера и текущее потребление ими памяти, я приведу только три:
container_memory_usage_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",container="POD",container_name="POD",id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e619e5dc53ed9efcef63f5fe1d7ee71.slice/docker-b6549e892baa8687e4e98a106024b5c31a4af077d7c5544af03a3c72ec8997e0.scope",image="k8s.gcr.io/pause:3.1",instance="controlplane",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="controlplane",kubernetes_io_os="linux",name="k8s_POD_etcd-controlplane_kube-system_0e619e5dc53ed9efcef63f5fe1d7ee71_0",namespace="kube-system",pod="etcd-controlplane",pod_name="etcd-controlplane"} 45056
container_memory_usage_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",container="POD",container_name="POD",id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a815a40_f2de_11ea_88d2_0242ac110032.slice/docker-76711789af076c8f2331d8212dad4c044d263c5cc3fa333347921bd6de7950a4.scope",image="k8s.gcr.io/pause:3.1",instance="controlplane",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="controlplane",kubernetes_io_os="linux",name="k8s_POD_kube-proxy-nhzhn_kube-system_5a815a40-f2de-11ea-88d2-0242ac110032_0",namespace="kube-system",pod="kube-proxy-nhzhn",pod_name="kube-proxy-nhzhn"} 45056
container_memory_usage_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",container="POD",container_name="POD",id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6473aeea_f2de_11ea_88d2_0242ac110032.slice/docker-24ef0e898e1bb7dec9854b67291171aa9c5715d7683f53bdfc2cef49a19744fe.scope",image="k8s.gcr.io/pause:3.1",instance="node01",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="node01",kubernetes_io_os="linux",name="k8s_POD_kube-proxy-6v49x_kube-system_6473aeea-f2de-11ea-88d2-0242ac110032_0",namespace="kube-system",pod="kube-proxy-6v49x",pod_name="kube-proxy-6v49x"} 835584
Выставим метку, которая содержится в метриках, чтобы отфильтровать один: container_memory_usage_bytes{container_name="prometheus"}
container_MEMORY_usage_bytes{beta_Kubernetes_io_arch="amd64",beta_Kubernetes_io_os="linux",container="prometheus",container_name="prometheus",id="/kubePODs.slice/kubePODs-burstable.slice/kubePODs-burstable-PODeaf4e833_f2de_11ea_88d2_0242ac110032.slice/Docker-b314fb5c4ce8894f872f05bdd524b4b7d6ce5415aeb3fb91d6048441c47584a6.scope",image="sha256:b82ef1f3aa072922c657dd2b2c6b59ec0ac88e69c447998291066e1f67e741d8",instance="node01",JOB="Kubernetes-cadvisor",Kubernetes_io_arch="amd64",Kubernetes_io_hostname="node01",Kubernetes_io_os="linux",name="k8s_prometheus_prometheus-5b77b7d695-knf44_istio-system_eaf4e833-f2de-11ea-88d2-0242ac110032_0",namespace="istio-system",POD="prometheus-5b77b7d695-knf44",POD_name="prometheus-5b77b7d695-knf44"}
283443200
Приведём в Mb: container_memory_usage_bytes {container_name="prometheus"} / 1000 / 1000
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",container="prometheus",container_name="prometheus",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf4e833_f2de_11ea_88d2_0242ac110032.slice/docker-b314fb5c4ce8894f872f05bdd524b4b7d6ce5415aeb3fb91d6048441c47584a6.scope",image="sha256:b82ef1f3aa072922c657dd2b2c6b59ec0ac88e69c447998291066e1f67e741d8",instance="node01",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="node01",kubernetes_io_os="linux",name="k8s_prometheus_prometheus-5b77b7d695-knf44_istio-system_eaf4e833-f2de-11ea-88d2-0242ac110032_0",namespace="istio-system",pod="prometheus-5b77b7d695-knf44",pod_name="prometheus-5b77b7d695-knf44"}
286.18752
Отфильтруем по container_memory_usage_bytes{container_name="prometheus", instance="node01"}
beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",container="prometheus",container_name="prometheus",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf4e833_f2de_11ea_88d2_0242ac110032.slice/docker-b314fb5c4ce8894f872f05bdd524b4b7d6ce5415aeb3fb91d6048441c47584a6.scope",image="sha256:b82ef1f3aa072922c657dd2b2c6b59ec0ac88e69c447998291066e1f67e741d8",instance="node01",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="node01",kubernetes_io_os="linux",name="k8s_prometheus_prometheus-5b77b7d695-knf44_istio-system_eaf4e833-f2de-11ea-88d2-0242ac110032_0",namespace="istio-system",pod="prometheus-5b77b7d695-knf44",pod_name="prometheus-5b77b7d695-knf44"}
289.890304
А на второй его нет: container_memory_usage_bytes{container_name="prometheus", instance="node02"}
no data
Есть и агрегатные функции sum(container_memory_usage_bytes) / 1000 / 1000 / 1000
{} 22.812798976
max(container_memory_usage_bytes) / 1000 / 1000 / 1000
{} 3.6422983679999996
min(container_memory_usage_bytes) / 1000 / 1000 / 1000
{} 0
Можно и сгруппировать по меткам instance: max(container_memory_usage_bytes) by (instance) / 1000 / 1000 / 1000
{instance="controlplane"} 1.641836544
{instance="node01"} 3.6622745599999997
Можно производить действия с однотипными метками и отфильтровывать: container_memory_mapped_file / container_memory_usage_bytes * 100 > 80
{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",container="POD",container_name="POD",id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45f10af1ae684722cbd74cb11807900.slice/docker-5cb2f2083fbc467b8b394b27b69686d309f951450bcb910d509572aea9922806.scope",image="k8s.gcr.io/pause:3.1",instance="controlplane",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="controlplane",kubernetes_io_os="linux",name="k8s_POD_kube-controller-manager-controlplane_kube-system_e45f10af1ae684722cbd74cb11807900_0",namespace="kube-system",pod="kube-controller-manager-controlplane",pod_name="kube-controller-manager-controlplane"}
80.52631578947368
Посмотреть на метрики файловой системы можно с помощью container_fs_limit_bytes, который выдаёт большой список приведу несколько из него:
container_fs_limit_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",container="POD",container_name="POD",device="/dev/vda1",id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e619e5dc53ed9efcef63f5fe1d7ee71.slice/docker-b6549e892baa8687e4e98a106024b5c31a4af077d7c5544af03a3c72ec8997e0.scope",image="k8s.gcr.io/pause:3.1",instance="controlplane",job="kubernetes-cadvisor",kubernetes_io_arch="amd64",kubernetes_io_hostname="controlplane",kubernetes_io_os="linux",name="k8s_POD_etcd-controlplane_kube-system_0e619e5dc53ed9efcef63f5fe1d7ee71_0",namespace="kube-system",pod="etcd-controlplane",pod_name="etcd-controlplane"}