下载chart

helm pull oci://registry-1.docker.io/bitnamicharts/kube-prometheus --version 8.30.1

下载模版

https://grafana.com/grafana/dashboards/15758-kubernetes-views-namespaces/?tab=revisions

https://grafana.com/grafana/dashboards/1860-node-exporter-full/?tab=revisions

安装

prometheus-values.yaml 文件如下：

global:
  imageRegistry: ""
  imagePullSecrets: []
  storageClass: ""

fullnameOverride: "kube-prometheus"

operator:
  image:
    registry: docker.io
    repository: bitnami/prometheus-operator
    tag: 0.72.0-debian-12-r1
    pullPolicy: IfNotPresent
    pullSecrets: []
  resources: {}

prometheus:
  enabled: true
  image:
    registry: docker.io
    repository: bitnami/prometheus
    tag: 2.50.1-debian-12-r1
    pullSecrets: []
  persistence:
    enabled: true
    size: 10Gi
  resources: {}
  replicaCount: 1
  podDisruptionBudget:
    enabled: true
  serviceMonitor:
    enabled: true
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
    hostname: prometheus.zili.work
  retention: 5d
  thanos:
    create: false

alertmanager:
  image:
    registry: docker.io
    repository: bitnami/alertmanager
    tag: 0.27.0-debian-12-r1
    pullSecrets: []

  persistence:
    enabled: false
    accessModes:
      - ReadWriteOnce
    size: 5Gi

  resources: {}

  replicaCount: 1

  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
    hostname: alert.prometheus.zili.work

## 
# node-exporter:
# kube-state-metrics:
# ServiceMonitors:

coreDns:
  enabled: true

kubelet:
  enabled: true
  replicaCount: 1

blackboxExporter:
  enabled: true
  replicaCount: 1

kubeApiServer:
  enabled: true

kubeControllerManager:
  enabled: true
  namespace: kube-system
  endpoint:
    - 192.168.2.133
  service:
    enabled: true
    ports:
      http: 10257
    targetPorts:
      http: 10257
  serviceMonitor:
    https: true
    insecureSkipVerify: true

kubeScheduler:
  enabled: true
  namespace: kube-system
  endpoint:
    - 192.168.2.133
  service:
    enabled: true
    ports:
      http: 10259
    targetPorts:
      http: 10259
  serviceMonitor:
    https: true
    insecureSkipVerify: true

kubeProxy:
  enabled: true
  namespace: kube-system
  endpoint:
    - 192.168.2.133
    - 192.168.2.135
    - 192.168.2.135

helm upgrade --install kube-prometheus . -f prometheus-values.yaml -n prometheus --create-namespace

集群外实例添加

additional-scrape-configurations

1. 新建一个配置文件, 如prometheus-additional.yaml

- job_name: 
  static_configs:
  - targets: ["192.168.1.32:9100"]

这里的配置写法，和维护 prometheus的启动配置文件一致。

2. 为配置文件创建一个secret

生成配置

kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml --dry-run=client -oyaml > additional-scrape-configs.yaml

应用配置

# 注意namespace 保持一致
kubectl apply -f additional-scrape-configs.yaml -n prometheus

3. 在编排中，添加这个配置

prometheus:
  ...
  ...
  additionalScrapeConfigs:
    enabled: true
    type: external
    external:
      name: additional-scrape-configs
      key: prometheus-additional.yaml

更新服务即可

FAQ

unhealthy

1. 是否为pod，非pod的服务，需指定endpotin、port、tls等
2. 网络策略（防火墙、服务端口等）
3. RBAC 授权

我的集群为kubeadm安装，默认这几个服务的端口绑定在localhost 127.0.0.1上，需要将地址暴露出来，这里为了方便，统一调整为 0.0.0.0

配置变更后，服务会自动重启，无须额外操作。

sed -e "s/- --bind-address=127.0.0.1/- --bind-address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-controller-manager.yaml
sed -e "s/- --bind-address=127.0.0.1/- --bind-address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-scheduler.yaml

更改编排文件，端口如下，再次执行即可。

kubeControllerManager:
  enabled: true
  namespace: kube-system
  endpoint:
    - 192.168.2.133
  service:
    enabled: true
    ports:
      http: 10257
    targetPorts:
      http: 10257
  serviceMonitor:
    https: true
    insecureSkipVerify: true
kubeScheduler:
  enabled: true
  namespace: kube-system
  endpoint:
    - 192.168.2.133
  service:
    enabled: true
    ports:
      http: 10259
    targetPorts:
      http: 10259
  serviceMonitor:
    https: true
    insecureSkipVerify: true

proxy的调整

编辑configmap， 修改metricsBindAddress的值为0.0.0.0,重启proxy服务即可