Monitoring Your Kubernetes Ingress with Ambassador, Prometheus, and Grafana

When you move to the cloud and start deploying applications onto Kubernetes, one of the first challenges you have to overcome is getting user traffic into your cluster.ence this is why we created the Ambassador Edge Stack API gateway. The next set of challenges, however, often relate to being able to understand what is happening within your system during steady-state operation and also, critically, what is occurring when things are going wrong.

This is all about making your system observable, and implementing observability often starts at the edge of your system, where the user begins their interaction with your applications. We’ve already written about how Ambassador supports distributed tracing, and how you can view ingress logs, and in this article, you will learn about how to collect and visualize metrics with Prometheus and Grafana.

Deployment

This guide will focus on deploying Prometheus and Grafana alongside Ambassador in Kubernetes using the Prometheus Operator.

Note: Both Prometheus and Grafana can be deployed as standalone applications outside Kubernetes. This process is well documented in the documentation for their individual projects.

Ambassador & Envoy

Ambassador makes it easy to output Envoy-generated statistics to Prometheus.

Starting with Ambassador 0.71.0, Prometheus can scrape statistics directly from Envoy’s /metrics endpoint, which removes the need to configure Ambassador to output statistics to StatsD, which was previously the case.

Suppose you are upgrading from an older version of Ambassador. In that case, it’s worth noting that the statistics scraped from the /metrics endpoint are not the same as those scraped from StatsD, and you will need to configure Envoy to output statistics to a separate collector before being scraped by Prometheus. You can use the Prometheus StatsD Exporter to do this.

1. Deploy the StatsD Exporter in the `default` namespace

kubectl apply -f https://getambassador.io/yaml/monitoring/statsd-sink.yaml

2. Configure Ambassador to output statistics to statsd

In the Ambassador deployment, add the `STATSD_ENABLED` and `STATSD_HOST` environment variables to tell Ambassador where to output statistics.

...
        env:
        - name: AMBASSADOR_NAMESPACE
          valueFrom:
          fieldRef:
            fieldPath: metadata.namespace
       - name: STATSD_ENABLED
         value: "true"
       - name: STATSD_HOST
         value: "statsd-sink.default.svc.cluster.local"
...

This will configure Ambassador to output statistics to the Prometheus StatsD exporter.

Prometheus Operator

The Prometheus Operator for Kubernetes provides an easy way to manage your Prometheus deployment using Kubernetes-style resources with custom resource definitions (CRDs).

1. Deploy the Prometheus Operator

To deploy the Prometheus Operator, you can clone the repository and follow the instructions in the README, or simply apply the published YAML with kubectl .

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/bundle.yaml

2. Deploy Prometheus by creating a Prometheus CRD.

First, create RBAC resources for your Prometheus instance

kubectl apply -f https://www.getambassador.io/yaml/monitoring/prometheus-rbac.yaml

Then, copy the YAML below and save it in a file called prometheus.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  type: ClusterIP
  ports:
  - name: web
    port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    prometheus: prometheus
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  ruleSelector:
    matchLabels:
      app: prometheus-operator
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      app: ambassador
  resources:
    requests:
      memory: 400Mi

Create the Prometheus instance with kubectl :

kubectl apply -f 

3. Create a ServiceMonitor

Finally, we need to tell Prometheus where to scrape metrics from. The Prometheus Operator easily manages this using a ServiceMonitor CRD. To tell Prometheus to scrape metrics from Ambassador’s /metrics endpoint, copy the following YAML to a file called ambassador-monitor.yaml and apply it with kubectl.

If you are running a version of Ambassador higher than 0.71.0 and want to scrape metrics directly from the /metrics endpoint running in the default namespace:

--- 
apiVersion: monitoring.coreos.com/v1 
kind: ServiceMonitor 
metadata:   
  name: ambassador-monitor   
  namespace: monitoring   
  labels:     
    app: ambassador 
spec:   
  namespaceSelector:     
    matchNames:     
    - default   
  selector:     
    matchLabels:       
      service: ambassador-admin   
  endpoints:   
  - port: ambassador-admin

If you are scraping metrics from a statsd-sink deployment:

--- 
apiVersion: monitoring.coreos.com/v1 
kind: ServiceMonitor 
metadata:   
  name: statsd-monitor   
  namespace: monitoring   
  labels:     
    app: ambassador 
spec:   
  namespaceSelector:     
    matchNames:     
    - default   
  selector:     
    matchLabels:       
      service: statsd-sink   
  endpoints:   
  - port: prometheus-metrics

At this point, Prometheus is now correctly configured to gather metrics from Ambassador.

Helm

You can also use helm to install Prometheus via the Prometheus Operator. The default helm chart will install Prometheus and configure it to monitor your Kubernetes cluster.

This section will focus on setting up Prometheus to scrape statistics from Ambassador, and it’s worth noting that the configuration of the helm chart and analysis of statistics from other cluster components is outside of the scope of this documentation.

1. Install the Prometheus Operator from the helm chart

helm install -n prometheus stable/prometheus-operator

2. Create a ServiceMonitor

The Prometheus Operator Helm chart creates a Prometheus instance that is looking for ServiceMonitors with label: release=prometheus.

If you are running a version higher than 0.71.0 and want to scrape metrics directly from the /metrics endpoint of Ambassador running in the default namespace:

--- 
apiVersion: monitoring.coreos.com/v1 
kind: ServiceMonitor 
metadata:   
  name: ambassador-monitor   
  namespace: monitoring   
  labels:     
    release: prometheus 
spec:   
  namespaceSelector:     
    matchNames:     
    - default   
  selector:     
    matchLabels:       
      service: ambassador-admin   
  endpoints:   
  - port: ambassador-admin

If you are scraping metrics from a statsd-sink deployment:

--- 
apiVersion: monitoring.coreos.com/v1 
kind: ServiceMonitor 
metadata:   
  name: statsd-monitor   
  namespace: monitoring   
  labels:     
    release: prometheus 
spec:   
  namespaceSelector:     
    matchNames:     
    - default   
  selector:     
    matchLabels:       
      service: statsd-sink  
  endpoints:   
  - port: prometheus-metrics

Prometheus is now configured to gather metrics from Ambassador.

Notes on the Prometheus Operator

The Prometheus Operator creates a series of Kubernetes Custom Resource Definitions (CRDs) for managing Prometheus in Kubernetes.

+============================+=====================================+
| Custom Resource Definition | Description                         | 
+============================+=====================================+
| AlertManager               | An AlertManager handles alerts sent |
|                            | by the Prometheus server.           | 
+----------------------------+-------------------------------------+
| PrometheusRule             | Registers altering and reporting    |
|                            | rules with Prometheus.              |
+----------------------------+-------------------------------------+
| Prometheus                 | Creates a Prometheus instance.      |
+----------------------------+-------------------------------------+
| ServiceMonitor             | Tells Prometheus where to scrape    |
|                            | metrics from.                       |
+----------------------------+-------------------------------------+

CoreOS has published a full API reference to these different CRDs.

Grafana

Grafana is an open source graphing tool for plotting data points. Grafana allows you to create dynamic dashboards for monitoring your ingress traffic statistics collected from Prometheus.

We have published a sample dashboard you can use for monitoring your ingress traffic. Since the statistics from the /metrics and /stats endpoints are different, you will see a section in the dashboard for each use case.

Note: If you deploy the Prometheus Operator via the helm chart, a Grafana dashboard is created by default. You can use this dashboard or set grafana.enabled: false and follow the instructions below.

To deploy Grafana behind Ambassador: replace {{AMBASSADOR_IP}} with the IP address of your Ambassador service, copy the YAML below, and apply it with kubectl:

--- 
apiVersion: extensions/v1beta1 
kind: Deployment
metadata: 
  name: grafana   
  namespace: default 
spec: 
  replicas: 1 
  template: 
    metadata: 
      labels: 
        app: grafana 
        component: core
    spec: 
      containers: 
      - image: grafana/grafana:6.2.0 
        name: grafana-core 
        imagePullPolicy: IfNotPresent 
        resources: 
          limits: 
            cpu: 100m 
            memory: 100Mi 
          requests: 
            cpu: 100m 
            memory: 100Mi 
        env: 
          - name: GR_SERVER_ROOT_URL 
            value: {{AMBASSADOR_IP}}/grafana 
          - name: GF_AUTH_BASIC_ENABLED 
            value: "true" 
          - name: GF_AUTH_ANONYMOUS_ENABLED 
            value: "false" 
        readinessProbe: 
          httpGet: 
            path: /login 
            port: 3000 
        volumeMounts: 
        - name: grafana-persistent-storage 
          mountPath: /var 
      volumes: 
      - name: grafana-persistent-storage 
        emptyDir: {}

Now, create a service and Mapping to expose Grafana behind Ambassador:

--- 
apiVersion: v1 
kind: Service 
metadata: 
  name: grafana 
  namespace: monitoring 
  annotations: 
    getambassador.io/config: | 
      --- 
      apiVersion: ambassador/v1 
      kind: Mapping 
      name: grafana-mapping 
      prefix: /grafana/ 
      service: grafana.monitoring 
spec: 
  ports: 
    - port: 80 
      targetPort: 3000 
  selector: 
    app: grafana 
    component: core

Now, access Grafana by going to {AMBASSADOR_IP}/grafana/ and logging in with username: admin : password: admin.

Import the provided dashboard by clicking the plus sign in the left side-bar, clicking New Dashboard in the top left, selecting Import Dashboard, and entering the dashboard ID(10434).

Viewing Statistics

So far you have created an environment where you have Ambassador deployed as an API gateway, Prometheus scraping and collecting metrics from Ambassador, and a Grafana dashboard to view these metrics.

You can easily view a sample of these statistics via the Grafana dashboard at {AMBASSADOR_IP}/grafana/, and logging in with the credentials above

The example dashboard you installed above displays statistics about the API response codes, number of connections, connection length, and number of registered services.

To view the full set of statistics available to Prometheus, you can access the Prometheus UI by running:

kubectl port-forward -n monitoring service/prometheus 9090

and going to http://localhost:9090/ from a web browser.

In the UI, you can click the dropdown to see all of the stats Prometheus is able to scrape from Ambassador.

The Prometheus data model is, at its core, time-series based. Therefore, it makes it easy to represent rates, averages, peaks, minimums, and histograms. Review the Prometheus documentation for a full reference on how to work with this data model.

Next Steps

With this tutorial complete, you can now begin creating a richer monitoring and observability experience at your system’s edge, based on your specific requirements. I always recommend implementing baseline metrics, such as the Golden Signals, as defined by the Google SRE team, but after you have these covered, you will inevitably want to implement other signals that are important to your business.

If you have any questions about Ambassador Labs please join our Slack, drop us a line in the comments below, or @ambassadorlabs on Twitter.