Using a9s Messaging

This topic describes how developers use a9s Messaging.

Use a9s Messaging With an App

To use a9s Messaging with an application, create a service instance and bind the service instance to your application. For more information on managing service instances, see Managing Service Instances with the cf CLI.

View the a9s Messaging Service

After the tile is installed, you can see the a9s-messaging and its service plans appear in your PCF marketplace. Run cf marketplace to see the service listing:

$ cf marketplace
Getting services from marketplace in org test / space test as admin...
OK
service         plans                                             description
a9s-messaging36  rabbitmq-single-small, rabbitmq-cluster-small,    This is the a9s Messaging 36 service.
                rabbitmq-single-big, rabbitmq-cluster-big

Create a Service Instance

To provision a RabbitMQ database, run cf create-service. For example:

$ cf create-service a9s-messaging36 rabbitmq-single-small my-rabbitmq-service

Depending on your infrastructure and service broker utilization, it might take several minutes to create the service instance.

Run the cf services command to view the creation status. This command displays a list of all your service instances. To view the status of a specific service instance, run cf service NAME-OF-YOUR-SERVICE.

Bind an Application to a Service Instance

After your database is created, run cf bind-service to bind the service to your application:

$ cf bind-service a9s-messaging-app my-rabbitmq-service

Restage or Restart Your Application

To enable your appliation to access the service instance, run cf restage or cf restart to restage or restart your app.

Obtain Service Instance Access Credentials

After a service instance is bound to an application, the credentials of your RabbitMQ database are stored in the environment variables of the application. Run cf env APP-NAME to display the environment variables.

You can find the credentials in the VCAP_SERVICES key.

$ cf env a9s-messaging-app
Getting env variables for app a9s-messaging-app in org test / space test as admin...
OK

System-Provided:
{
  "VCAP_SERVICES": {
    "a9s-messaging37": [
      {
        "credentials": {
          "host": "hostname.node.dcx.consul",
          "hosts": [
            "hostname.node.dcx.consul"
          ],
          "password": "password",
          "port": 5672,
          "http_api_uri": "http://username:password@hostname.node.dcx.consul",
          "http_api_uris": ["http://username:password@hostname.node.dcx.consul"],
          "protocols": {
            "amqp": {
              "host": "hostname.node.dcx.consul",
              "hosts": [
                "hostname.node.dcx.consul"
              ],
              "password": "password",
              "port": 5672,
              "ssl": false,
              "uri": "amqp://username:password@hostname.node.dcx.consul:5672",
              "username": "username"
            },
            "management": {
              "username": "username",
              "password": "password",
              "path": "/api",
              "ssl": false,
              "host": "hostname.node.dcx.consul",
              "hosts": [
                "hostname.node.dcx.consul"
              ],
              "uri": "http://username:password@hostname.node.dcx.consul",
              "uris": ["http://username:password@hostname.node.dcx.consul"]
            }
          },
          "ssl": false,
          "uri": "username:password@hostname.node.dcx.consul:5672",
          "username": "username"
        },
        "label": "a9s-messaging37",
        "name": "rabbitmq-app",
        "plan": "rabbitmq-single-small",
        "provider": null,
        "syslog_drain_url": null,
        "tags": [
         "messaging",
         "queue",
         "rabbitmq"
        ],
        "volume_mounts": []
      }
    ]
  }
}
...

You can use the host, username and password values to connect to your database with a RabbitMQ client.

Protocols Hash

The credentials hash contains a protocols hash that lists credentials by protocol. The credentials at the root level of the credentials are either for AMQP with non SSL service plans or AMQPS with SSL service plans.

Protocol keyEnabled when
amqpalways
amqp+sslSSL plan
mqttMQTT plugin enabled
mqtt+sslSSL plan and MQTT plugin enabled
stompSTOMP plugin enabled
stomp+sslSSL plan and STOMP plugin enabled
managementalways

Best Practices

There are some best practices for using service binding information in apps in a separate document.

Delete an a9s Messaging Service Instance

WARNING: Before deleting a service instance, you must backup data stored in your database. This operation cannot be undone and all the data is lost when the service instance is deleted.

Before you can delete a service instance, you must unbind it from all apps.

List Available Services

Run cf services to list your available services.

$ cf services

Getting services in org test / space test as admin...
OK

name                  service          plan                    bound apps         last operation
my-rabbitmq-service   a9s-messaging36   rabbitmq-single-small   a9s-messaging-app   create succeeded

This example shows that my-rabbitmq-service is bound to the a9s-messaging-app application.

Unbind a Service Instance

Run cf unbind-service to unbind the service instance from your app.

$ cf unbind-service a9s-messaging-app my-rabbitmq-service

Delete a Service Instance

After unbinding the service, it is no longer bound to an application. Run cf delete-service to delete the service:

$ cf delete-service my-rabbitmq-service

It may take several minutes to delete the service. Deleting a service deprovisions the corresponding infrastructure resources. Run the cf services command to view the deletion status.

Upgrade the Service Instance to Another Service Plan

Once created, you can upgrade your service instance to another, larger service plan. A larger service plan provides more CPU, RAM and storage. For more information, see the Update a Service Instance of the Managing Service Instances with the cf CLI topic.

$ cf update-service my-rabbitmq-service -p a-bigger-plan

Here are the plans you can upgrade to depending on the one you are currently using:

  • If you are currently using the rabbitmq-single-small plan, you can upgrade to the rabbitmq-single-big plan.

  • If you are currently using the rabbitmq-cluster-small plan, you can upgrade to the rabbitmq-cluster-big plan.

Add a Graphite Endpoint

If you want to monitor your service with Graphite, you can set an endpoint to where to information will be sent with the cf update-service command. This command expects the -c flag and a JSON string containing the graphite and metrics_prefix keys. Depending on your graphite provider the metrics_prefix might require that each metrics must start with an API key in their name. You can also change the interval within the data is send to the endpoint. Do to this modify interval the default is 10s.

$ cf update-service my-rabbitmq-service -c '{ "graphite": ["yourspace.your-graphite-endpoint.com:12345"], "metrics_prefix": "your-api-key.my-cluster-rabbitmq", "interval": "5"}'

RabbitMQ Metric Patterns

Following a list of the RabbitMQ metric patterns which will be streamed to a collocated Prometheus endpoint. The metrics can be divided into the groups:

  • Node Metrics
  • Queue Metrics
  • Port Checks
  • Ghost Queues

Node Metrics

*.rabbitmq.*.*.*.*.node.io_read_count
*.rabbitmq.*.*.*.*.node.disk_free
*.rabbitmq.*.*.*.*.node.disk_free_details.rate
*.rabbitmq.*.*.*.*.node.io_read_avg_time
*.rabbitmq.*.*.*.*.node.io_read_avg_time_details.rate
*.rabbitmq.*.*.*.*.node.io_read_bytes
*.rabbitmq.*.*.*.*.node.io_read_bytes_details.rate
*.rabbitmq.*.*.*.*.node.io_read_count
*.rabbitmq.*.*.*.*.node.io_read_count_details.rate
*.rabbitmq.*.*.*.*.node.io_sync_avg_time
*.rabbitmq.*.*.*.*.node.io_sync_avg_time_details.rate
*.rabbitmq.*.*.*.*.node.io_write_avg_time
*.rabbitmq.*.*.*.*.node.io_write_avg_time_details.rate
*.rabbitmq.*.*.*.*.node.mem_used
*.rabbitmq.*.*.*.*.node.mem_used_details.rate
*.rabbitmq.*.*.*.*.node.sockets_used
*.rabbitmq.*.*.*.*.node.fd_total
*.rabbitmq.*.*.*.*.node.fd_used
*.rabbitmq.*.*.*.*.node.sockets_total
*.rabbitmq.*.*.*.*.node.mem_limit
*.rabbitmq.*.*.*.*.node.disk_free_limit
*.rabbitmq.*.*.*.*.node.proc_total
*.rabbitmq.*.*.*.*.node.proc_used
*.rabbitmq.*.*.*.*.node.uptime
*.rabbitmq.*.*.*.*.node.partitioned

Queue Metrics

*.rabbitmq.*.*.*.*.queue.<queue-name>.memory
*.rabbitmq.*.*.*.*.queue.<queue-name>.messages
*.rabbitmq.*.*.*.*.queue.<queue-name>.messages_details.rate
*.rabbitmq.*.*.*.*.queue.<queue-name>.messages_ready
*.rabbitmq.*.*.*.*.queue.<queue-name>.messages_ready_details.rate
*.rabbitmq.*.*.*.*.queue.<queue-name>.messages_unacknowledged
*.rabbitmq.*.*.*.*.queue.<queue-name>.messages_unacknowledged_details.rate
*.rabbitmq.*.*.*.*.queue.<queue-name>.consumers
*.rabbitmq.*.*.*.*.queue.<queue-name>.messages_ram
*.rabbitmq.*.*.*.*.queue.<queue-name>.messages_ready_ram
*.rabbitmq.*.*.*.*.queue.<queue-name>.messages_unacknowledged_ram
*.rabbitmq.*.*.*.*.queue.<queue-name>.messages_persistent
*.rabbitmq.*.*.*.*.queue.<queue-name>.message_bytes
*.rabbitmq.*.*.*.*.queue.<queue-name>.message_bytes_ready
*.rabbitmq.*.*.*.*.queue.<queue-name>.message_bytes_unacknowledged
*.rabbitmq.*.*.*.*.queue.<queue-name>.message_bytes_ram
*.rabbitmq.*.*.*.*.queue.<queue-name>.message_bytes_persistent
*.rabbitmq.*.*.*.*.queue.<queue-name>.disk_reads
*.rabbitmq.*.*.*.*.queue.<queue-name>.disk_writes
*.rabbitmq.*.*.*.*.queue.<queue-name>.backing_queue_status.q1
*.rabbitmq.*.*.*.*.queue.<queue-name>.backing_queue_status.q2
*.rabbitmq.*.*.*.*.queue.<queue-name>.backing_queue_status.q3
*.rabbitmq.*.*.*.*.queue.<queue-name>.backing_queue_status.q4
*.rabbitmq.*.*.*.*.queue.<queue-name>.backing_queue_status.len
*.rabbitmq.*.*.*.*.queue.<queue-name>.backing_queue_status.next_seq_id
*.rabbitmq.*.*.*.*.queue.<queue-name>.backing_queue_status.avg_ingress_rate
*.rabbitmq.*.*.*.*.queue.<queue-name>.backing_queue_status.avg_egress_rate
*.rabbitmq.*.*.*.*.queue.<queue-name>.backing_queue_status.avg_ack_ingress_rate
*.rabbitmq.*.*.*.*.queue.<queue-name>.backing_queue_status.avg_ack_egress_rate

Port Checks

*.rabbitmq.*.*.*.*.port_checks.amqp_port_open
*.rabbitmq.*.*.*.*.port_checks.http_api_port_open
*.rabbitmq.*.*.*.*.port_checks.web_stomp_port_open
*.rabbitmq.*.*.*.*.port_checks.stomp_port_open
*.rabbitmq.*.*.*.*.port_checks.epmd_port_open
*.rabbitmq.*.*.*.*.port_checks.mqtt_client_port_open
*.rabbitmq.*.*.*.*.port_checks.mqtt_client_tls_port_open
*.rabbitmq.*.*.*.*.port_checks.amqp_tls_port_open
*.rabbitmq.*.*.*.*.port_checks.web_mqtt_port_open
*.rabbitmq.*.*.*.*.port_checks.stomp_tls_port_open

Ghost Queues

The idea behind this metric is to support the detection of ghost queues. Ghost queues are a known rabbitmq-bug/-artecfact, so ideally this metric shouldn't appear.

*.rabbitmq.*.*.*.*.ghost_queue.<ghost-queue-name>

Add a Syslog Endpoint

The cf update-service command used with the -c flag can let you stream your syslog to a third-party service. In this case, the command expects a JSON string containing the syslog key. You can also change the interval for the syslog with the same key than for the graphite endpoint interval.

$ cf update-service my-rabbitmq-service -c '{ "syslog": ["logs4.your-syslog-endpoint.com:54321"], "interval": "5" }'

Cloud Foundry Application Security Groups

This topic describes how to check whether a security group was created.

Each a9s Data Service will automatically create and update Cloud Foundry security groups in order to protected service instances to be accessed by applications not running in the same Cloud Foundry applications space. To get a better understanding about Security Groups you can have a look on the Understanding Application Security Groups topic.

Get Service Instance GUID

Run cf service INSTANCE_NAME --guid to get the guid of the service instance.

$ cf service my-rabbitmq --guid
ca16f111-5073-40b7-973a-156c75dd3028

Check available Security Groups

To see all available security groups use cf security-groups.

$cf security-groups
Getting security groups as demo@anynines.com
OK

     Name                                         Organization     Space
#0   public_networks
#1   dns
#2   tcp_open
#3   guard_432fb752-876d-443b-a311-a075f4df2237   demonstrations   demo
#4   guard_ca16f111-5073-40b7-973a-156c75dd3028   demonstrations   demo

There we can see a security group with the name guard_ca16f111-5073-40b7-973a-156c75dd3028 was successfully created.

NOTE: in some circumstances the connection between the application and the service instance is not possible, in this case check if a security group was created.

Backup and Restore Service Instances

a9s Messaging provides an easy way to create backups and restore if needed. For a more detailed description, please see the a9s Service Dashboard documentation.

Use RabbitMQ Plugins

a9s Messaging allows to enable the following RabbitMQ plugins:

  • rabbitmq_consistent_hash_exchange
  • rabbitmq_federation
  • rabbitmq_federation_management
  • rabbitmq_mqtt
  • rabbitmq_sharding
  • rabbitmq_shovel
  • rabbitmq_shovel_management
  • rabbitmq_stomp
  • rabbitmq_tracing
  • rabbitmq_event_exchange
    • Available in a9s Messaging 3.7 only

Plugin Activation

You can activate RabbitMQ plugins with the cf create-service and cf update-service commands using additional configuration parameters.

$ cf create-service a9s-messaging36 rabbitmq-single-small my-rabbitmq-service -c '{ "plugins": ["rabbitmq_shovel", "rabbitmq_shovel_management"] }'
$ cf update-service my-rabbitmq-service -c '{ "plugins": ["rabbitmq_shovel", "rabbitmq_shovel_management"] }'

Plugins for Protocols

Some plugins that can be enabled with a9s Messaging provide additional protocols for use with RabbitMQ. The usernames and passwords for these protocols are the same as with the default ACMQ protocol. Protocols enabled via plugin will be set to the default RabbitMQ vhost.

ProtocalTCP PortSSL PortFurther information
MQTT18838883RabbitMQ MQTT Docs
STOMP6161361614RabbitMQ STOMP Docs

Create RabbitMQ Users

a9s Messaging allows to create new users with different roles (permissions) by creating service keys.

The possible roles are:

  • management
  • policymaker
  • monitoring
  • administrator

You can read more about RabbitMQ roles and permissions here.

Create a New Role

You can create a new user with specific permissions (roles) giving the role following custom parameter. This parameter receives an array of roles:

$ cf create-service-key my-rabbitmq-service my-key -c '{"roles": ["administrator", "management"]}'

Use RabbitMQ Management Dashboard

a9s Messaging has management dashboard support enabled. The dashboard is running on the service instance VM so it is not possible to open the dashboard in your browser directly.

Option: via (Go) router

When the platform operator has configured the (Go)router to route the RabbitMQ Management Dashboard the dashboard will be exposed under a URL formatted as https://service-instance-guid.a9s-messaging-dashboard.system.domain.

Obtaining the Dashboard URL

The dashboard URL that provides access to the a9s dashboard contains all the information required, to create a URL that will access the RabbitMQ Dashboard. This command will create the URL you need:

cf service a9s-messaging-serive-instance | grep dashboard | sed  's/^.*https:\/\/\(.*\)\/service-instances\/\(.*\)/https:\/\/\2.\1/'

You can use this URL to navigate to the dashboard login page in your web browser.

Obtaining Credentials

To log into the RabbitMQ dashboard you will need a username and a password. If you want to see the dashboard from the perspective of your app, you could use the apps credentials. Accessing the credential used by your app is simple:

cf env your-app-name

If you would prefer to have a separate RabbitMQ user for accessing the dashboard you can create and use a service key:

cf create-service-key a9s-messaging-service-instance dashboard
cf service-key a9s-messaging-service-instance dashboard

Option: with CF SSH

CF provides a smart way to create SSH forward tunnels via a pushed application.

Make sure that SSH access to the app is enabled. You can enable SSH access with cf enable-ssh MY-AWESOME-APP. If you do not have permission, contact your administrator.

For more information about this feature see the Accessing Apps with SSH section of the CF documentation.

First of all you must have an application bound to the service. How to do this see Bind an Application to a Service Instance.

NOTE: cf ssh support must be enabled in the platform. Ask your administrator if you are not sure.

Get Dashboard Url and Credentials

When you follow this instructions Obtain Service Instance Access Credentials you will get the hostname of the service and the user credentials.

$ cf env rabbitmq-app
Getting env variables for app rabbitmq-app in org phartz / space develop as admin...
OK

System-Provided:
{
 "VCAP_SERVICES": {
  "a9s-messaging36": [
   {
    "credentials": {
     "host": "d67901c.service.dc1.a9svs",
     "username": "a9s-brk-usr",
     "password": "a9s-password",
     "port": 5672,
     "uri": "amqp://a9s-brk-usr:a9s-password@d67901c.service.dc1.a9svs:5672"
    },
    "label": "a9s-messaging36",
    "name": "myrabbit",
    "plan": "rabbitmq-single-small",
    "provider": null,
    "syslog_drain_url": null,
    "tags": [
     "message broker"
    ],
    "volume_mounts": []
  }
  ]
 }
}

Notice the host d67901c.service.dc1.a9svs, the username a9s-brk-usr and the password a9s-password. You will need this in the next step.

Create Tunnel to The Management Dashboard

With the cf ssh as mentioned before you can create a ssh forward tunnel to the management dashboard.

$ cf ssh rabbitmq-app -L 15672:d67901c.service.dc1.a9svs:15672
vcap@956aaf4e-6da9-4f69-4b1d-8e631a403312:~$

NOTE: Don't forget to close the session with exit.

Login to The Management Dashboard

When the ssh tunnel is open you can access the Dashboard with your Browser at http://localhost:15672.

To login to the management dashboard you need the user credentials. management-dashboard-login

If the login was successful, you should see the dashboard as followed. management-dashboard

Using HA Queues in Cluster Setup

If clusters are enrolled, this does not mean that all created queues are HA queues!

By default there are two policies defined to allow HA queues and no HA queues. These policies are depending on the queue name.

If the queue name starts with ha. the queue is a HA queue and consists of clusternodecount/2 + 1 queues (one master queue and the rest mirror queues).

This guarantees that in the case of a network partition, the queue is still available in the bigger partition. Additionally, it allows to use most of performance of the clusternodes, by reducing replication and queues.

Additionally there is another policy for not important queues, this policy is for all queues starting with noha., in this case the queue is not mirrored and if the node which hold this queue goes down, the queue is also not available anymore.

Furter information can be found here: [2] You can also find information about self created policies.

Using custom policies in RabbitMQ

Important: The a9s Messaging service configures by default two policies named ha and noha. These policies will always be reconfigured on a restart of a RabbitMQ service instance. Therefore they shouldn't be changed manually.

To create and use custom policies, please see Parameters and Policies — RabbitMQ

Make a Service Instance Locally Available

It is possible to access any of the a9s Data Services locally. That means you can connect with a local client to the service for any purpose such as debbuging. CF provides a smart way to create SSH forward tunnels via a pushed application. For more information about this feature see the Accessing Apps with SSH section of the CF documentation.

First of all you must have an application bound to the service. How to do this see Bind an Application to a Service Instance.

NOTE: cf ssh support must be enabled in the platform. Ask your administrator if you are not sure.

Get The Service Url and Credentials

When you follow this instructions Obtain Service Instance Access Credentials you will get the hostname of the service and the user credentials.

$ cf env a9s-messaging-app
Getting env variables for app a9s-messaging-app in org test / space test as admin...
OK

System-Provided:
{
  "VCAP_SERVICES": {
   "a9s-messaging": [
    {
      "credentials": {
       "host": [
        "d67901c.service.dc1.a9svs"
       ],
       "password": "a9s-brk-usr",
       "username": "a9s-password"
     },
     "label": "a9s-messaging",
     "name": "my-rabbitmq-service",
     "plan": "rabbitmq-cluster-small"
    }
   ]
  }
}
...

Notice the host d67901c.service.dc1.a9svs, the username a9s-brk-usr and the password a9s-password. You will need this in the next step.

Create a Tunnel to The Service

With the cf ssh as mentioned before you can create a ssh forward tunnel to the management dashboard. Use port 9200 to connect to the a9s Messaging Instance.

$ cf ssh a9s-messaging-app -L 9200:d67901c.service.dc1.a9svs:9200
vcap@956aaf4e-6da9-4f69-4b1d-8e631a403312:~$

When the ssh tunnel is open you can access the instance over the address localhost:9200.

NOTE: Don't forget to close the session with exit.

Create a Fork of a Service Instance

The procedure of forking a service instance involves creating a backup of a service instance and restoring it to a different service instance.

Having two service instances is a prerequisite for the process:

$ cf s
Getting services in org jebreuer_anynines_com / space test as jebreuer@anynines.com...

name      service          plan                     bound apps   last operation
rabbit1   a9s-messaging37   rabbitmq-cluster-small   bindingo     create succeeded
rabbit2   a9s-messaging37   rabbitmq-cluster-small                create succeeded

Additional prerequisites regarding command line tools:

  • BASH (some shell)
  • cat
  • openssl
  • python (see below for the verion required)
  • node (tested with v6.11.0)

Open the service dashboard of the service instance you want to fork. We use rabbit1 for this example. You can find the dashboard URL like this:

$ cf service rabbit1
Showing info of service rabbit1 in org jebreuer_anynines_com / space test as jebreuer@anynines.com...

name:            rabbit1
service:         a9s-messaging37
bound apps:      bindingo
tags:
plan:            rabbitmq-cluster-small
description:     This is a service creating and managing dedicated RabbitMQ service instances, powered by the anynines Service Framework
documentation:
dashboard:       https://a9s-messaging-dashboard.de.a9s.eu/service-instances/950cb675-3ed9-4613-8bb6-b2d618391d2f

[...]

Make sure you set a encryption password for the backups using the service instance dashboard (Change Backup Settings). Create a backup using the dashboard. Download the backup to your local machine. The filename will be something like racsd92baee-1522222422893. Decrypt the backup and write its contents to a file:

$ cat racsd92baee-1522222422893 | openssl enc -aes256 -d -pass 'pass:mytopsecretpassword' | gunzip -c > backup_settings.json

Create a tunnel to the admin interface (like explained in a previous chapter) of the RabbitMQ instance that will be the fork of the original instance. I used rabbit1 as the original instance and rabbit2 as the fork instance. The tunnel is needed to get the matching version of the rabbitmqadmin script as well as to restore the backed up data. I assume you're using something like this to set up the tunnel:

$ cf ssh someapp -L 127.0.0.1:15672:racsd92baee.service.dc1.a9ssvc:15672

racsd92baee.service.dc1.a9ssvc is the host of the rabbit2 service instance.

Go to http://127.0.0.1:15672/cli/ to download the rabbitmqadmin tool. rabbitmqadmin is a python script. On that page you'll find information about the python version required.

Download a copy of the backup script restore_queues.js. Make sure to chmod u+x the script.

Restore the backed up queues using the restore script, e.g.:

./restore_queues.js $(which python) ~/Downloads/rabbitmqadmin 127.0.0.1 15672 a9s-brk-usr-xxxxxxxx xxxxxxxyyyyyyyyyzzzzzzzzz ./backup_settings.json

Setup Disk Usage Alerts

Each service comes with the a9s Parachute. This component monitors ephemeral and persistent disk usage. See the a9s Parachute documentation how to configure the component.