Kube Prometheus

Kube Prometheus provides an easy-to-operate end-to-end Kubernetes cluster monitoring solution using Prometheus together with PrometheusOperator.

Kube Prometheus provides an easy-to-operate end-to-end Kubernetes cluster monitoring solution using Prometheus together with PrometheusOperator.

Kube Prometheus includes the following packages:

Dependencies

Prometheus Operator

Install

You can install kube-prometheus in a few different ways, but we recomend to use the Master Helmfile.

Install using Master Helmfile

To install kube-prometheus run:

Install kube-prometheus

helmfile -f /conf/kops/helmfile.yaml --selector namespace=monitoring,chart=kube-prometheus sync

These are some of the environment variables you may want to configure:

Environment variables can be specified in the Geodesic Module’s Dockerfile or using Chamber storage, which is recommended for all secrets.

Install using Custom Helmfile

Add to your Kubernetes Backing Services Helmfile this code

helmfile.yaml

repositories:
- name: "coreos-stable"
  url: "https://s3-eu-west-1.amazonaws.com/coreos-charts/stable"

releases:
- name: "prometheus-operator"
  namespace: "kube-system"
  labels:
    chart: "prometheus-operator"
    component: "monitoring"
    namespace: "kube-system"
    vendor: "coreos"
    default: "true"
  chart: "coreos-stable/prometheus-operator"
  version: "0.0.23"
  set:
    - name: "rbacEnable"
      value: "false"

    - name: "jobLabel"
      value: "prometheus-operator"

    - name: "image.repository"
      value: "quay.io/coreos/prometheus-operator"

    - name: "image.tag"
      value: 'v0.17.0'

    - name: "image.pullPolicy"
      value: "IfNotPresent"

    - name: "resources.limits.cpu"
      value: "10m"

    - name: "resources.limits.memory"
      value: "64Mi"

    - name: "resources.requests.cpu"
      value: "5m"

    - name: "resources.requests.memory"
      value: "32Mi"

    - name: "global.hyperkube.repository"
      value: "quay.io/coreos/hyperkube"

    - name: "global.hyperkube.tag"
      value: 'v1.7.6_coreos.0'

    - name: "global.hyperkube.pullPolicy"
      value: "IfNotPresent"

    - name: "prometheusConfigReloader.repository"
      value: "quay.io/coreos/prometheus-config-reloader"

    - name: "prometheusConfigReloader.tag"
      value: 'v0.0.3'

    - name: "configmapReload.repository"
      value: "quay.io/coreos/configmap-reload"

    - name: "configmapReload.tag"
      value: 'v0.0.1'

    - name: "kubeletService.enable"
      value: "true"

    - name: "kubeletService.namespace"
      value: "kube-system"

    - name: "kubeletService.name"
      value: "kubelet"

Then follow the instructions for running helmfile sync.

Extending Functionality

You can extend the default functionality of kube-prometheus to collect custom metrics and send alerts based on the metrics, and display the metric in Grafana charts.

Custom Netrics Collection

  1. Check the list of available exporters to make sure there isn’t already an Exporter that will meet your needs. For example, there are exporters that will work with JMX out-of-the-box.
  2. If there is no exporter, then you’ll need to write your own exporter that will provide required metrics in correct format
  3. Create a Kubernetes service that provides an exporter http endpoint. We recommend then deploying this service using a Helm Chart.
  4. Register the exporter for service discovery by creating a Kubernetes resource of type ServiceMonitor that points to the service created in step 3.

Examples

Custom Dashboards using Grafana

Before you start, read the basic introduction about grafana dashboards and panels

The next steps depend on the way you installed kube-prometheus

Installation using Master Helmfile

grafana:
  serverDashboardFiles:
    dashboard_unique_file_name.json: |-
      EXPORTED CONTENT GOES HERE

Installation using Custom Helmfile

grafana:
  serverDashboardFiles:
    dashboard_unique_file_name.json: |-
      EXPORTED CONTENT GOES HERE
values:
  - "value.yaml"