Skip to main content
Version: 51.0.0

Using a9s Prometheus

This chapter describes how to use a9s Prometheus.

Use a9s Prometheus with an App

To use a9s Prometheus, you can follow the procedures in the create a Service Instance and a service key sections. For more information on managing Service Instances, see Managing Service Instances with the cf CLI.

View the a9s Prometheus Service

After the service is installed, a9s-prometheus and its service plans appear in your CF marketplace. You can run the cf marketplace command to see the service listing:

$ cf marketplace
Getting services from marketplace in org test / space test as admin...
OK

service plans description
a9s-prometheus prometheus-single-small, prometheus-single-medium, prometheus-single-big This is the a9s Prometheus service.

See the next section for instructions on creating a9s Prometheus Service Instances based on the plans listed in the cf marketplace output.

Create a Service Instance

To provision a Prometheus system, run cf create-service. For example:

cf create-service a9s-prometheus prometheus-single-small my-prometheus-service

Depending on your infrastructure and service broker usage, it may take several minutes to create the Service Instance.

Check the creation status using cf services. This displays a list of all your Service Instances. To check the status of a specific Service Instance, run cf service NAME-OF-YOUR-SERVICE.

Bind an Application to a Service Instance

After you created your Service Instance, run cf bind-service to bind the Service Instance to your application:

cf bind-service a9s-prometheus-app my-prometheus-service

Restage or Restart Your Application

To enable your app to access the Service Instance, run cf restage or cf restart to restage or restart your app.

Obtain Service Instance Access Credentials

After a Service Instance is bound to an application, the credentials of your Prometheus system are stored in the environment variables of the application. Run cf env APP-NAME to display the environment variables.

You can find the credentials in the VCAP_SERVICES key.

$ cf env a9s-prometheus-app
Getting env variables for app a9s-prometheus-app in org test / space test as admin...
OK

System-Provided:
{
"VCAP_SERVICES": {
"a9s-prometheus": [
{
"credentials": {
"alertmanager_urls": [
"EXAMPLE_ALERTMANAGER_URL"
],
"grafana_urls": [
"EXAMPLE_GRAFANA_URL"
],
"graphite_exporter_port": 9109,
"graphite_exporters": [
"EXAMPLE_GRAPHITE_URL"
],
"password": "EXAMPLE-PASSWORD",
"prometheus_urls": [
"EXAMPLE_PROMETHEUS_URL"
],
"username": "EXAMPLE-USERNAME"
},
"label": "a9s-prometheus",
"name": "my-prometheus-service",
"plan": "prometheus-single-small",
"tags": [
"metrics",
"logging",
"alerting",
"monitoring",
"prometheus"
]
}
]
}
}
...

You can use the alertmanager_urls, grafana_urls, prometheus_urls, username and password values to connect to the different APIs within your application.

Best Practices

There are some best practices for using service binding information in apps in a separate document.

Stream a9s Service Instance Metrics to a9s Prometheus

To use an a9s Prometheus Service Instance to monitor another a9s Service Instance, follow the first two steps of Use a9s Prometheus with an App.

Create a Service Key

After the a9s Prometheus Service Instance has been created, run cf create-service-key my-prometheus-service my-service-key to create a new service key for your Service Instance:

$ cf create-service-key my-prometheus-service my-service-key
Creating service key my-service-key for service instance my-prometheus-service as admin...
OK

To retrieve the required information from your service key, run $ cf service-key my-prometheus-service my-service-key:

$ cf service-key my-prometheus-service my-service-key
Getting key my-service-key for service instance my-prometheus-service as admin...

{
"alertmanager_urls": [
"http://d00a2c-alertmanager-0.node.dc1.consul:9093/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/alertmanager/"
],
"grafana_urls": [
"http://d00a2c-grafana-0.node.dc1.consul:3000/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/grafana/"
],
"graphite_exporter_port": 9109,
"graphite_exporters": [
"d00a2c-prometheus-0.node.dc1.consul"
],
"password": "a9s70e5d011f061afff3e4af8073aad09d3a058511a",
"prometheus_urls": [
"http://d00a2c-prometheus-0.node.dc1.consul:9090/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/prometheus/"
],
"username": "a9s-brk-usr-1f4553994f7a617f23ed4efa629ea7199e2a44d0"
}

Please note the fields graphite_exporters and graphite_exporters_port which will be used in the next step.

Update a a9s Service Instance

The cf update-service command used with the -c flag can let you stream your metrics to a third-party service. In this case, the command expects a JSON string containing the graphite key. This key contains the combination of the host and port where the Graphite metrics should be sent to.

You can retrieve this information from the previously created service key of your a9s Prometheus Service Instance, you need the values from the fields graphite_exporters and graphite_exporters_port.

$ cf update-service service-instance-to-monitor -c '{"graphite": ["d00a2c-prometheus-0.node.dc1.consul:9109"]}'

For more a9s Data Service specific information, like the available metrics, metrics frequency, etc., see the following pages:

See the a9s Service Instance Metrics

Once your monitored Service Instance is streaming into your a9s Prometheus you can use the Prometheus Dashboard to see the streamed metrics.

Stop Streaming a9s Service Instance Metrics

Follow the steps below to stop streaming Service Instance metrics to a9s Prometheus.

Overwrite Your Service Instance Configuration

If you want to stop streaming your Service Instance metrics to your a9s Prometheus Service Instance, you can overwrite the graphite key of your Service Instance configuration.

Run cf update-service INSTANCE-NAME -c PARAMETERS-AS-JSON to update the Graphite endpoint of the Service Instance you don't want to monitor anymore. For this, you need to overwrite the value of the graphite key with an empty array.

$ cf update-service service-instance-to-monitor -c '{"graphite": []}'

Delete the Service Key and Service Instance

You may want to delete the service key and the a9s Prometheus Service Instance itself, if you are not using it anymore. Run cf delete-service-key INSTANCE-NAME SERVICE-KEY-NAME and cf delete-service INSTANCE-NAME to do so.

$ cf delete-service-key my-prometheus-service my-service-key
Really delete the service key my-service-key?> y
Deleting service key my-service-key for service instance my-prometheus-service as admin...
OK

$ cf delete-service my-prometheus-service

It may take several minutes to delete the Service Instance. Deleting a Service Instance deprovisions the corresponding infrastructure resources. Run the cf services command to view the deletion status.

Stream Application Metrics to a9s Prometheus

Prometheus is designed to aggregate scraped data from various targets into its time series database. So, it can be as simple as implementing a metrics endpoint and allowing Prometheus to query that endpoint.

Due to network restrictions depending on the cloud environment, scraping targets can't be used easily. Therefore we will have a look at different options to stream your application metrics to an a9s Prometheus Service Instance.

Using a Custom Scrape Target

If your application implements a metrics endpoint and there are no network restrictions in your cloud environment, you can configure your application as a scrape target in your a9s Prometheus Service Instance. For more information see Scrape Configs.

Using the Graphite Exporter

a9s Prometheus contains a Graphite Exporter endpoint to which you can stream metrics. By default, the Graphite Exporter has rules defined to map Graphite compliant metrics to Prometheus compliant metrics with labels.

Additionally, via the strict matching settings, the Graphite Exporter filters metrics which don't match these rules. In order to allow the a9s Prometheus Service Instance to receive metrics other than the configured ones, you have to switch off the strict matching settings. For more information see here.

Create a Service Key to get Information about your Service Instance

Create a service key to get URLs of your a9s Prometheus Service Instance:

$ cf create-service-key my-prometheus-service my-key
$ cf service-key my-prometheus-service my-key

{
"alertmanager_urls": [
"http://d00a2c-alertmanager-0.node.dc1.consul:9093/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/alertmanager/"
],
"grafana_urls": [
"http://d00a2c-grafana-0.node.dc1.consul:3000/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/grafana/"
],
"graphite_exporter_port": 9109,
"graphite_exporters": [
"d00a2c-prometheus-0.node.dc1.consul"
],
"password": "a9s70e5d011f061afff3e4af8073aad09d3a058511a",
"prometheus_urls": [
"http://d00a2c-prometheus-0.node.dc1.consul:9090/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/prometheus/"
],
"username": "a9s-brk-usr-1f4553994f7a617f23ed4efa629ea7199e2a44d0"
}

Configure your application to stream its metric to the graphite_exporters endpoint and port graphite_exporter_port.

Using a Sidecar

Instead of using the Graphite Exporter directly in your application, you can also use a Sidecar that scrapes the metrics endpoint of your application and streams them to the Graphite Exporter. For more information see Using a Sidecar to Collect Application Metrics With a9s Prometheus.

See the Application Metrics

Once your monitored application is streaming into your a9s Prometheus Service Instance you can use the Prometheus Dashboard to see the streamed metrics.

Stop Streaming Application Metrics to a9s Prometheus

Depending on the approach you are using to stream your application metrics to your a9s Prometheus Service Instance, you have to execute different steps.

Stop Using a Custom Scrape Target

If you want to stop your a9s Prometheus Service Instance from scraping your application, you have to update your Service Instance and remove your application as a target from the scrape configs.

$ cf update-service my-prometheus-service -c '{ "scrape_configs": [] }'

Stop Using the Graphite Exporter

If you want to stop your application from streaming its metrics to your a9s Prometheus Service Instance, you have to change the configuration of your application and remove both the graphite_exporters endpoint and the graphite_exporter_port port.

You can also delete the service key that contains the properties and credentials you used in your application.

$ cf delete-service-key my-prometheus-service my-key

Stop Using a Sidecar

If you are using the Sidecar approach, you have to unbind your application from your a9s Prometheus Service Instance.

$ cf unbind-service my-app my-prometheus-service

Access the a9s Prometheus Dashboards

You have to execute the following command to get the required information to access the dashboards of your a9s Prometheus Service Instance:

cf service my-prometheus-service

you will get an output similar to this:

Showing info of service my-prometheus-service in org test / space test as admin...

name: my-prometheus-service
service: a9s-prometheus
tags:
plan: single-small
description: This is a service creating and managing dedicated Prometheus to monitor applications and Service Instances, powered by the anynines Service Framework
documentation: https://docs.anynines.com
dashboard: https://a9s-dashboard.system.a9s-ds-concourse.a9s-ops.de/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f
service broker: prometheus-service-broker

Showing status of last operation from service SERVICE_NAME...

status: update succeeded
message:
started: 2022-01-20T08:48:11Z
updated: 2022-01-20T08:49:18Z

There are no bound apps for this service.

Extract the dashboard URL from the command's output, in the example above:

https://a9s-dashboard.system.a9s-ds-concourse.a9s-ops.de/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f

Grafana Dashboard

To access the Grafana dashboard, you have to append grafana to the URL:

https://a9s-dashboard.system.a9s-ds-concourse.a9s-ops.de/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/grafana

For more information on the usage of the Grafana Dashboard see the Grafana Dashboard page.

Prometheus Dashboard

To access the Prometheus dashboard, you have to append prometheus to the URL:

https://a9s-dashboard.system.a9s-ds-concourse.a9s-ops.de/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/prometheus

The Prometheus dashboard allows you to visualize the streamed/scraped metrics.

See the Streamed Metrics

For the following example, we will use an a9s PostgreSQL as the Service Instance to monitor.

  1. Open the Prometheus Dashboard, which should redirect to the Graph page:

  2. Here you can query a specific metric, or select one from the Metrics Explorer:

  3. After executing the query you can see both the full query itself (which also shows the job that contains the metric) and its resulting graph:

Alertmanager Dashboard

To access the Alertmanager dashboard, you have to append alertmanager to the URL:

https://a9s-dashboard.system.a9s-ds-concourse.a9s-ops.de/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/alertmanager

Delete an a9s Prometheus Service Instance

danger

This operation cannot be undone and all the data is lost when the Service Instance is deleted.

Before you can delete a Service Instance, you must unbind it from all apps and delete all service keys.

List Bound Applications

Run cf services to list your available services and the bound application to them.

$ cf services

Getting services in org test / space test as admin...

name service plan bound apps last operation
my-prometheus-service a9s-prometheus prometheus-single-small a9s-prometheus-app create succeeded

This example shows that my-prometheus-service is bound to the a9s-prometheus-app app.

Unbind a Service Instance

Run cf unbind to unbind the service from your app.

cf unbind-service a9s-prometheus-app my-prometheus-service

List Service Keys

Run cf service-keys my-prometheus-service to list the service keys created for your Service Instance.

$ cf service-keys my-prometheus-service

Getting keys for service instance my-prometheus-service as admin...

name
my-service-key

This example shows that my-prometheus-service has one service key my-service-key.

Delete a Service Key

Run cf delete-service-key my-prometheus-service my-service-key to delete the service key.

$ cf delete-service-key my-prometheus-service my-service-key

Deleting key my-service-key for service instance my-prometheus-service as admin...
OK

Delete a Service Instance

After unbinding the Service Instance from the bound applications and deleting the related service key, it can now be deleted. Run cf delete-service to delete the Service Instance:

cf delete-service my-prometheus-service

It may take several minutes to delete the Service Instance. Deleting a Service Instance deprovisions the corresponding infrastructure resources. Run the cf services command to view the deletion status.

Upgrade the Service Instance to another Service Plan

You can update the service plan of your Service Instance by executing the following command:

cf update-service my-prometheus-service -p a-similar-newer-plan
caution

a9s Prometheus does NOT support Service Instance upgrades from Service Instances containing Grafana 5 to any other type of a9s Prometheus Service Instance.

Cloud Foundry Application Security Groups

This topic describes how to check whether an Application Security Group (ASG) was created.

Each a9s Data Service will automatically create and update Cloud Foundry ASGs in order to protect Service Instances to be accessed by applications not running in the same Cloud Foundry applications space. For further details on Security Groups you can refer to the Understanding Application Security Groups page in the official Cloud Foundry documentation.

Get Service Instance GUID

Run cf service INSTANCE_NAME --guid to get the guid of the Service Instance.

$ cf service my-prometheus-service --guid
2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f

Check available Security Groups

To see all available security groups use cf security-groups.

$ cf security-groups
Getting security groups as admin
OK

name organization space lifecycle
#0 public_networks
#1 dns
#3 a9s-asg-432fb752-876d-443b-a311-a075f4df2237 demonstrations demo running
a9s-asg-432fb752-876d-443b-a311-a075f4df2237 demonstrations demo staging
#4 a9s-asg-2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f demonstrations demo running
a9s-asg-2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f demonstrations demo staging

There we can see a security group named a9s-asg-2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f was successfully created.

note

In some circumstances it is not possible to bind/connect the application and the a9s Prometheus Service Instance, in such cases you must verify if a security group was created.

Make a Service Instance Locally Available

It is possible to access any of the a9s Data Services locally. That means you can connect with a local client to the Service Instance for any purpose such as debugging. CF provides a smart way to create SSH forward tunnels via a pushed application. For more information about this feature see the Accessing Apps with SSH section of the CF documentation.

First of all you must have an application bound to the Service Instance. You can find a step-by-step guide on how to do this in the Bind an Application to a Service Instance section.

note

cf ssh support must be enabled in the platform. Ask your administrator if you are not sure.

Get The Service URL and Credentials

Follow the instructions how to Obtain Service Instance Access Credentials and you will get the user credentials and the accessible URLs of your Service Instance.

$ cf env a9s-prometheus-app
Getting env variables for app a9s-prometheus-app in org test / space test as admin...
OK

System-Provided:
{
"VCAP_SERVICES": {
"a9s-prometheus": [
{
"credentials": {
"alertmanager_urls": [
"http://d37f7da-alertmanager-0.node.dc1.consul:9093/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/alertmanager"
],
"grafana_urls": [
"http://d37f7da-grafana-0.node.dc1.consul:3000/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/grafana"
],
"graphite_exporter_port": 9109,
"graphite_exporters": [
"d37f7da-prometheus-0.node.dc1.consul"
],
"password": "a9s-password",
"prometheus_urls": [
"http://d37f7da-prometheus-0.node.dc1.consul:9090/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/prometheus"
],
"username": "a9s-brk-usr"
},
"label": "a9s-prometheus",
"name": "my-prometheus-service",
"plan": "prometheus-single-small"
}
]
}
}
...

Notice the URLs alertmanager_urls, grafana_urls and prometheus_urls, the username a9s-brk-usr and the password a9s-password. You will need these URLS and credentials in the next step.

Create a Tunnel to The Service Instance

With the cf ssh, as mentioned before, you can create a ssh forward tunnel to your Service Instance. To connect to the Prometheus API you have to execute the following command:

$ cf ssh a9s-prometheus-app -L 9090:d37f7da-prometheus-0.node.dc1.consul:9090
vcap@956aaf4e-6da9-4f69-4b1d-8e631a403312:~$

Once the SSH tunnel is open you can access the Prometheus API via your browser or the use of cURL through the URL localhost:9090/service-instances/2fe38eeb-e3a0-4d18-86bc-1c0c0b0e303f/prometheus, this will require you to provide the above mentioned credentials.

You can access both the Grafana API and the Alertmanager API of your Service Instance in a similar manner.

note

Don't forget to close the session with exit.

a9s Prometheus Custom Parameters

As an Application Developer you have the opportunity to customize your Service Instance by using custom parameters. Custom parameters are passed on to a Service Instance by using the -c switch of the cf CLI commands cf create-service and cf update-service.

For example,

cf update-service my-prometheus-service -c '{ "my_custom_parameter": "value" }'

would set the custom parameter named my_custom_parameter to the value value for the Service Instance my-prometheus-service.

You don't have to utilize those settings. There are sane defaults in place that fit your service plan well.

mapping_strict_match

This property is used to configure the setting mapping-strict-match of the Graphite exporter. The default value is true.

If mapping_strict_match is set to true, only the metrics specified in the Graphite mapping configuration are allowed. Thus metrics that don't match the mapping configuration are skipped. This is useful if you have a very large set of metrics. For more information please see the Graphite Documentation.

scrape_configs

info

This custom parameter is disabled by default. Please contact your Platform Operator for further details.

The scrape_configs parameter allows the Application Developer to configure scraping targets for your a9s Prometheus Service Instance in the form of a valid JSON object.

note

Please keep in mind that while scraping targets can be set, the default targets are always set and take precedence over the ones provided by the custom parameter.

The example below showcases the minimum configuration required in order to properly set a scraping target. For a comprehensive list of accepted parameters please refer to the Prometheus documentation:

{
"scrape_configs": [
{
"job_name": "example_metrics",
"scrape_interval": "60s",
"scrape_timeout": "30s",
"static_configs": [
{
"targets": [
"example-app-prometheus-hungry-hippopotamus-ow.apps.example.a9s-dummy.de"
]
}
]
}
]
}
$ cf update-service my-prometheus-service -c custom_scrape_configs.json
scrape_config JSON values

While there is a validation in place to ensure that the provided JSON object is valid, it should be noted that the contents themselves are not validated. Thus, it is the responsibility of the Application Developer to properly sanitize and validate this input.