Using a9s Prometheus

Provision an a9s Prometheus Service Instance

To provision an a9s Prometheus instance follow the instructions below.

View the a9s Prometheus Service

View the Service Offering

After the tile is installed, run cf marketplace to see a9s-prometheus service offering and its service plans:

$ cf marketplace
Getting services from marketplace in org test / space test as admin...
OK

service            plans                                                                                              description
a9s-prometheus     prometheus-single-small, prometheus-single-medium, prometheus-single-big   This is the a9s Prometheus service.

Create a Service Instance

To provision a Prometheus database, run cf create-service. For example:

cf create-service a9s-prometheus prometheus-single-small my-prometheus-service

Depending on your infrastructure and service broker usage, it may take several minutes to create the service instance.

You can check the creation status of your instances by running cf services. This displays a list of all your service instances. To check the status of a specific service instance, you can run cf service NAME-OF-YOUR-SERVICE.

Stream Service Instance Metrics to a9s Prometheus

To use a9s Prometheus with an a9s PostgreSQL service instance, please see the Add a Graphite Endpoint section in the a9s PostgreSQL application developer documentation.

Stream Application Metrics to a9s Prometheus

Prometheus is designed to aggregate scraped data from various targets into its time serious databases. Due to network restrictions depending on the cloud environment scraping targets can't be used easily. a9s Prometheus contains a Graphite Exporter endpoint where you can stream metrics to. By default the Graphite Exporter has rules defined to map Graphite compliant metrics to Prometheus compliant metrics with labels. Also the Graphite exporter filters metrics which doesn't match to these metrics. In order to allow the a9s Prometheus instance to receive other metrics then the configured ones you have to switch off the strict matching. You can do that by following the documentation.

Create a Service Key to get Information about your deployment

Create a service key to get URLs of your Prometheus deployment:

$ cf create-service-key my-prometheus-service my-key
$ cf service-key my-prometheus-service my-key

{
  "graphite_exporters": ["d37f7da-prometheus-0.node.dc1.consul"],
  "graphite_exporter_port": 9109,
  "prometheus_urls": "http://d37f7da-prometheus-0.node.dc1.consul:9090/",
  "grafana_urls": "http://d37f7da-grafana-0.node.dc1.consul:3000/",
  "alertmanager_urls": "http://d37f7da-alertmanager-0.node.dc1.consul:9093/",
  "username": "username",
  "password": "password"
}

Configure your application to stream its metric to the graphite_exporters endpoint and port graphite_exporter_port.

Access the a9s Prometheus Dashboards

To access the dashboards of your a9s Prometheus instance, execute the following command:

cf service SERVICE_NAME

you will get an output similar to this:

Showing info of service SERVICE_NAME in org ORG / space SPACE as admin...

name:             SERVICE_NAME
service:          a9s-prometheus
tags:
plan:             single-small
description:      This is a service creating and managing dedicated Prometheus to monitor applications and service instances, powered by the anynines Service Framework
documentation:    https://www.anynines.com
dashboard:        https://a9s-dashboard.system.a9s-ds-concourse.a9s-ops.de/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f
service broker:   prometheus-service-broker

Showing status of last operation from service SERVICE_NAME...

status:    update succeeded
message:
started:   2022-01-20T08:48:11Z
updated:   2022-01-20T08:49:18Z

There are no bound apps for this service.

Extract the dashboard URL from here, i.e.:

https://a9s-dashboard.system.a9s-ds-concourse.a9s-ops.de/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f

Grafana Dashboard

To access Grafana append grafana to the URL:

https://a9s-dashboard.system.a9s-ds-concourse.a9s-ops.de/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/grafana

Prometheus Dashboard

To access Prometheus append prometheus to the URL:

https://a9s-dashboard.system.a9s-ds-concourse.a9s-ops.de/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/prometheus

Alertmanager Dashboard

To access Alertmanager append alertmanager to the URL:

https://a9s-dashboard.system.a9s-ds-concourse.a9s-ops.de/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/alertmanager

We also plan to provide an easier solution to access the dashboards for the final release.

There are some best practices for using service binding information in apps in a separate document.

If you want to stop streaming your service instance metrics to your Prometheus instance, you can overwrite the graphite key of your service configuration.

Run

cf update-service INSTANCE-NAME -c PARAMETERS-AS-JSON

to update the graphite endpoint of the instance you don't want to monitor anymore. For this, you need to overwrite the value of the graphite key with an empty string.

cf update-service my-redis-service -c '{"graphite": []}'

Delete the Prometheus Service Instance

Run cf delete-service INSTANCE-NAME to delete the service instance:

cf delete-service my-prometheus-service

It may take several minutes to delete the service. Deleting a service deprovisions the corresponding infrastructure resources.

Run the cf services command to view the deletion status.

To come back to a clean setup, your should also reconfigure your service instance.

Delete the Service Key and Instance

If you are not using it anymore, you may want to delete the service key and the service instance itself. Run cf delete-service-key INSTANCE-NAME SERVICE-KEY-NAME and cf delete-service INSTANCE-NAME to do so.

cf delete-service-key my-prometheus-service my-key

cf delete-service my-prometheus-service

It may take several minutes to delete the service. Deleting a service deprovisions the corresponding infrastructure resources. Run the cf services command to view the deletion status.

Prometheus Custom Parameters

As an end user you have the opportunity to customize your service instance by using custom parameters. Custom parameters are passed on to a service instance by using the -c switch of the cf CLI commands cf create-service and cf update-service. For example

cf update-service my-instance -c '{ "mapping_strict_match": false }'

You don't have to utilize those settings. There are sane defaults in place that fit your service plan well.

Every parameter corresponds to a property in the configuration file for the respective Prometheus version.

mapping_strict_match

This property is used to configure the setting mapping-strict-match of the Graphite exporter. If this is set to true, only the metrics specified in the Graphite mapping configuration are allowed. Thus metrics that don't match the mapping configuration are skipped. This is useful if you have a very large set of metrics. For more information please see the Graphite Documentation.