本文參考

[k8s安裝prometheus并持久化數(shù)據(jù)_/prometheus-config-reloader:-CSDN博客](https://blog.csdn.net/vic_qxz/article/details/119598466)

• 前置要求: 已經(jīng)部署了NFS或者其他存儲(chǔ)的K8s集群.

  這里注意networkpolicies網(wǎng)絡(luò)策略問(wèn)題，可以后面刪除這個(gè)策略，這里可以查看我之前的文檔。

部署kube-prometheus

1. 這里是配置好才執(zhí)行這個(gè)，我們還沒(méi)有配置存儲(chǔ)什么的需要進(jìn)行修改

  $ git clone https://github.com/coreos/kube-prometheus.git  #版本最新的是0.13.0
$ kubectl create -f manifests/setup
$ until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl create -f manifests/  #如果資源已經(jīng)存在，則會(huì)報(bào)錯(cuò)
kubectl apply -f  跟這個(gè)一樣   #如果資源已經(jīng)存在，則會(huì)進(jìn)行更新詳解一下   
2. `kubectl create -f manifests/setup`: 使用 `kubectl` 命令創(chuàng)建 Kubernetes 資源，這些資源位于 manifests/setup 目錄下。一般來(lái)說(shuō)，這個(gè)命令會(huì)創(chuàng)建一些必要的資源，比如 ServiceAccount、ClusterRole 和 ClusterRoleBinding 等，用于配置 Prometheus 和 Grafana 在 Kubernetes 中的權(quán)限。3. `until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done`: 這是一個(gè)循環(huán)命令，它會(huì)持續(xù)執(zhí)行 `kubectl get servicemonitors --all-namespaces` 命令，直到能夠成功獲取到所有命名空間中的 ServiceMonitor 資源。ServiceMonitor 是 Prometheus Operator 中的一種資源類型，用于指定 Prometheus 服務(wù)器應(yīng)該如何監(jiān)控應(yīng)用程序。在這個(gè)命令中，通過(guò) `until` 循環(huán)檢查是否已經(jīng)創(chuàng)建了所有的 ServiceMonitor 資源。4. `kubectl create -f manifests/`: 使用 `kubectl` 命令創(chuàng)建 Kubernetes 資源，這些資源位于 manifests/ 目錄下。在這個(gè)命令中，一般會(huì)創(chuàng)建 Prometheus、Alertmanager、Grafana 等監(jiān)控相關(guān)的資源。

持久化數(shù)據(jù)我這里用的是NFS創(chuàng)建動(dòng)態(tài)的pv
我的storageclass名稱是nfs-storageclass

root@k8s-master01:~/test/prometheus/kube-prometheus-0.13.0# kubectl get sc
NAME                  PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage   fuseim.pri/ifs           Delete          Immediate              false                  4d20h
nfs-storageclass      prometheus-nfs-storage   Retain          Immediate              false                  16h

kube-prometheus的組件簡(jiǎn)介及配置變更

1.從整體架構(gòu)看，prometheus 一共四大組件。 exporter 通過(guò)接口暴露監(jiān)控?cái)?shù)據(jù)， prometheus-server 采集并存儲(chǔ)數(shù)據(jù)， grafana 通過(guò)prometheus-server查詢并友好展示數(shù)據(jù)， alertmanager 處理告警，對(duì)外發(fā)送

prometheus-operator

prometheus-operator 服務(wù)是deployment方式部署，他是整個(gè)基礎(chǔ)組件的核心，他監(jiān)控我們自定義的 prometheus 和alertmanager，并生成對(duì)應(yīng)的 statefulset。 就是prometheus和alertmanager服務(wù)是通過(guò)他部署出來(lái)的。

修改配置文件

grafana-pvc

創(chuàng)建grafana的存儲(chǔ)卷. 并修改grafana-deployment.yaml文件, 將官方的emptyDir更換為persistentVolumeClaim

1.創(chuàng)建pvc

$ cd kube-prometheus/manifests/
$ cat  grafana-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:# PersistentVolumeClaim 名稱name: grafananamespace: monitoringannotations:# 與 nfs-storageClass.yaml metadata.name 保持一致volume.beta.kubernetes.io/storage-class: "nfs-storageclass" 
spec:# 使用的存儲(chǔ)類為 nfs-storageclassstorageClassName: "nfs-storageclass"# 訪問(wèn)模式為 ReadWriteManyaccessModes:- ReadWriteMany#- ReadWriteOnceresources:# 存儲(chǔ)請(qǐng)求為 50Girequests:storage: 50Gi $ kubectl apply -f grafana-pvc.yaml

2.修改默認(rèn)的grafana配置文件

$ vim grafana-deployment.yaml...##找到 grafana-storage, 添加上面創(chuàng)建的pvc: grafana. 然后保存.volumes:- name: grafana-storagepersistentVolumeClaim:claimName: grafana
...$ kubectl apply -f grafana-deployment.yaml

prometheus-k8s持久化

prometheus-server 獲取各端點(diǎn)數(shù)據(jù)并存儲(chǔ)與本地，創(chuàng)建方式為自定義資源 crd中的prometheus。 創(chuàng)建自定義資源prometheus后，會(huì)啟動(dòng)一個(gè)statefulset，即prometheus-server. 默認(rèn)是沒(méi)有配置持久化存儲(chǔ)的

1.修改配置文件

$ cd kube-prometheus/manifests/
$ vim prometheus-prometheus.yaml  
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:labels:prometheus: k8sname: k8snamespace: monitoring
spec:alerting:alertmanagers:- name: alertmanager-mainnamespace: monitoringport: webstorage: #這部分為持久化配置volumeClaimTemplate:spec:storageClassName: nfs-23 accessModes: ["ReadWriteOnce"]resources:requests:storage: 100GinodeSelector:kubernetes.io/os: linuxpodMonitorNamespaceSelector: {}podMonitorSelector: {}replicas: 2resources:requests:memory: 400MiruleSelector:matchLabels:prometheus: k8srole: alert-rulessecurityContext:fsGroup: 2000runAsNonRoot: truerunAsUser: 1000serviceAccountName: prometheus-k8sserviceMonitorNamespaceSelector: {}serviceMonitorSelector: {}version: v2.17.2

執(zhí)行變更, 這里會(huì)自動(dòng)創(chuàng)建兩個(gè)指定大小的pv（prometheus-k8s-0，prometheus-k8s-1）

$ kubectl apply -f manifests/prometheus-prometheus.yaml 

修改存儲(chǔ)時(shí)長(zhǎng)

$ vim manifests/setup/prometheus-operator-deployment.yaml
....- args:- --kubelet-service=kube-system/kubelet- --logtostderr=true- --config-reloader-image=jimmidyson/configmap-reload:v0.3.0- --prometheus-config-reloader=quay.io/coreos/prometheus-config-reloader:v0.39.0- storage.tsdb.retention.time=180d   ## 修改存儲(chǔ)時(shí)長(zhǎng)
....
$ kubectl apply -f manifests/setup/prometheus-operator-deployment.yaml

添加ingress訪問(wèn)grafana和promethues

這里訪問(wèn)是有問(wèn)題的，參照我kubernetes-networkpolicies網(wǎng)絡(luò)策略問(wèn)題這篇文章解決

$ cat ingress.yml 
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:annotations:k8s.eip.work/workload: grafanak8s.kuboard.cn/workload: grafanageneration: 2labels:app: grafananame: grafananamespace: monitoring
spec:rules:- host: k8s-moni.fenghong.techhttp:paths:- backend:serviceName: grafanaservicePort: httppath: /
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:annotations:k8s.kuboard.cn/workload: prometheus-k8sgeneration: 2labels:app: prometheusprometheus: k8smanagedFields:- apiVersion: networking.k8s.io/v1beta1name: prometheus-k8snamespace: monitoring
spec:rules:- host: k8s-prom.fenghong.techhttp:paths:- backend:serviceName: prometheus-k8sservicePort: webpath: /

執(zhí)行apply

## 安裝 ingress controller
$ kubectl apply -f https://kuboard.cn/install-script/v1.18.x/nginx-ingress.yaml## 暴露grafana及prometheus服務(wù)
$ kubectl apply -f ingress.yml

web訪問(wèn)

配置kube-prometheus監(jiān)控額外的項(xiàng)目

添加additional-scrape-configs配置文件. 例如

$ cat monitor/add.yaml 
- job_name: 'prometheus'# metrics_path defaults to '/metrics'# scheme defaults to 'http'.static_configs:- targets: ['192.168.0.23:9100', '192.168.0.21:9101', '192.168.0.61:9100', '192.168.0.62:9100', '192.168.0.63:9100', '192.168.0.64:9100', '192.168.0.89:9100', '192.168.0.11:9100']
- job_name: 'mysql'static_configs:- targets: ['192.168.0.21:9104','192.168.0.23:9104']
- job_name: 'nginx'static_configs:- targets: ['192.168.0.23:9913']- job_name: 'elasticsearch'metrics_path: "/_prometheus/metrics"static_configs:- targets: ['192.168.0.31:9200']

創(chuàng)建secret文件, 我這里部署到了monitoring 命名空間.

$ kubectl create secret generic additional-scrape-configs --from-file=add.yaml --dry-run -oyaml  > additional-scrape-configs.yaml
$ kubectl apply -f additional-scrape-configs.yaml  -n monitoring

在prometheus-prometheus.yaml中添加 additionalScrapeConfigs 選項(xiàng).

$  cat prometheus-prometheus.yaml 
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:labels:prometheus: k8sname: k8snamespace: monitoring
spec:alerting:alertmanagers:- name: alertmanager-mainnamespace: monitoringport: webstorage: #這部分為持久化配置volumeClaimTemplate:spec:storageClassName: nfs-23 accessModes: ["ReadWriteOnce"]resources:requests:storage: 100Giimage: quay.io/prometheus/prometheus:v2.17.2nodeSelector:kubernetes.io/os: linuxpodMonitorNamespaceSelector: {}podMonitorSelector: {}replicas: 3 resources:requests:memory: 400MiruleSelector:matchLabels:prometheus: k8srole: alert-rulessecurityContext:fsGroup: 2000runAsNonRoot: truerunAsUser: 1000serviceAccountName: prometheus-k8sserviceMonitorNamespaceSelector: {}serviceMonitorSelector: {}version: v2.17.2additionalScrapeConfigs:name: additional-scrape-configskey: add.yaml

執(zhí)行apply即可

$ kubectl apply -f prometheus-prometheus.yaml

其他系統(tǒng)的訪問(wèn)

參考文檔

[Kube-prometheus部署Ingress為Prometheus-Grafana開(kāi)啟https_kube-prometheu配置ingress-CSDN博客](https://blog.csdn.net/Happy_Sunshine_Boy/article/details/107955691)

Prometheus

基于訪問(wèn)路徑過(guò)濾

修改yaml：kube-prometheus-0.5.0/manifests/prometheus-prometheus.yaml
在參數(shù)下：image: quay.io/prometheus/prometheus:v2.15.2，添加如下參數(shù)：
externalUrl: https://master170.k8s:30443/prometheus

kubectl apply -f prometheus-prometheus.yaml

配置：ingress-tls.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:kubernetes.io/ingress.class: nginxnginx.ingress.kubernetes.io/use-regex: "true"nginx.ingress.kubernetes.io/enable-cors: "true"nginx.ingress.kubernetes.io/rewrite-target: /$2name: prometheus-k8snamespace: monitoring
spec:rules:- host: #寫(xiě)你的域名http:paths:- path: /prometheus(/|$)(.*)pathType: ImplementationSpecificbackend:service:name: prometheus-k8sport:number: 9090

訪問(wèn)prometheus時(shí)，都要帶上“prometheus”：
舉例：
https://master170.k8s:30443/prometheus/graph

AlertManager

修改yaml：manifests/alertmanager-alertmanager.yaml
在參數(shù)下：image: quay.io/prometheus/alertmanager:v0.20.0，添加如下參數(shù)：
externalUrl: https://master170.k8s:30443/alertmanager

配置：ingress-tls.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:kubernetes.io/ingress.class: nginxnginx.ingress.kubernetes.io/use-regex: "true"nginx.ingress.kubernetes.io/enable-cors: "true"nginx.ingress.kubernetes.io/rewrite-target: /$2name: prometheus-k8snamespace: monitoring
spec:rules:- host: #寫(xiě)你的域名http:paths:#   - path: /prometheus(/|$)(.*)#     pathType: ImplementationSpecific#     backend:#       service:#         name: prometheus-k8s#         port:#           number: 9090- path: /alertmanager(/|$)(.*)pathType: ImplementationSpecificbackend:service:name: alertmanager-mainport:number: 9093  

訪問(wèn)alertmanager時(shí)，都要帶上“alertmanager”：
https://master170.k8s:30443/alertmanager/#/alerts