Using a9s Elasticsearch
This topic describes how developers use a9s Elasticsearch.
Use a9s Elasticsearch with an Application
To use a9s Elasticsearch with an application, create a service instance and bind the service instance to your application. For more information on managing service instances, see Managing Service Instances with the cf CLI.
View the a9s Elasticsearch Service
After the service is installed, you can see the a9s-elasticsearch
and its service plans appear in your CF marketplace. Run cf marketplace
to see the service listing:
$ cf marketplace
Getting services from marketplace in org test / space test as admin...
OK
service plans description
a9s-elasticsearch5 elasticsearch-single-small, elasticsearch-cluster-small, This is the a9s Elasticsearch 5 service.
elasticsearch-single-big, elasticsearch-cluster-big
Create a Service Instance
To provision an Elasticsearch database, run cf create-service
. For example:
cf create-service a9s-elasticsearch5 elasticsearch-single-small my-elasticsearch-service
Depending on your infrastructure and service broker utilization, it may take several minutes to create the service instance.
Run the cf services
command to view the creation status. This command displays a list of all your service instances. To view the status of a specific service instance, run cf service NAME-OF-YOUR-SERVICE
.
Bind an Application to a Service Instance
After your database is created, run cf bind-service
to bind the service to your application:
cf bind-service a9s-elasticsearch-app my-elasticsearch5-service
Restage or Restart Your Application
To enable your application to access the service instance, run cf restage
or cf restart
to restage or restart your application.
Obtain Service Instance Access Credentials
After a service instance is bound to an application, the credentials of your Elasticsearch database are stored in the environment variables of the application. Run cf env APP-NAME
to display the environment variables.
You can find the credentials in the VCAP_SERVICES key.
$ cf env a9s-elasticsearch-app
Getting env variables for app a9s-elasticsearch-app in org test / space test as admin...
OK
System-Provided:
{
"VCAP_SERVICES": {
"a9s-elasticsearch5": [
{
"credentials": {
"host": [
"EXAMPLE_HOST"
],
"hosts": [
"EXAMPLE_HOST"
],
"password": "EXAMPLE_USER",
"username": "EXAMPLE_PASSWORD",
"scheme": "http",
"port": 9200
},
"label": "a9s-elasticsearch5",
"name": "my-elasticsearch-service",
"plan": "elasticsearch-cluster-small",
"tags": [
"searchengine"
]
}
]
}
}
...
You can use the host, username and password values to connect to your database with a Elasticsearch client.
This Data Service does not have support for multiple unique service keys. Therefore, when creating services keys for this Data Service, every key will have the same set of credentials.
Best Practices
There are some best practices for using service binding information in apps in a separate document.
Delete an a9s Elasticsearch Service Instance
WARNING: Before deleting a service instance, you must backup data stored in your database. This operation cannot be undone and all the data is lost when the service is deleted.
Before you can delete a service instance, you must unbind it from all apps.
List Available Services
Run cf services
to list your available services.
$ cf services
Getting services in org test / space test as admin...
OK
name service plan bound apps last operation
my-elasticsearch-service a9s-elasticsearch5 elasticsearch-single-small a9s-elasticsearch-app create succeeded
This example shows that my-elasticsearch-service
is bound to the a9s-elasticsearch-app
application.
Unbind a Service Instance
Run cf unbind-service
to unbind the service from your application.
cf unbind-service a9s-elasticsearch-app my-elasticsearch-service
Delete a Service Instance
After unbinding the service, it is no longer bound to an application. Run cf delete-service
to delete the service:
cf delete-service my-elasticsearch-service
It may take several minutes to delete the service. Deleting a service deprovisions the corresponding infrastructure resources.
Run the cf services
command to view the deletion status.
Upgrade the Service Instance to another Service Plan
Once created, you can upgrade your service instance to another, larger service plan. A larger service plan provides more CPU, RAM and storage. For more information, see the Update a Service Instance of the Managing Service Instances with the cf CLI topic.
cf update-service my-elasticsearch-service -p a-bigger-plan
Add a Graphite Endpoint
Important: Streaming of logs and metrics might not be availabale for your plan! If unsure, please check your plan description.
If you want to monitor your service with Graphite, you can set the custom
parameter graphite
. It expects the host and port where the Graphite metrics
should be sent to.
For example, in order to send Graphite metrics to an endpoint
yourspace.your-graphite-endpoint.com:12345
, you can use the following
command:
cf update-service my-instance -c '{ "graphite": "yourspace.your-graphite-endpoint.com:12345" }'
The endpoint would then receive metrics in the format:
<service_guid>.<service_type>.<host>.<metric> <metric value> <metric timestamp>
Metrics Frequency
By default, metrics will be emitted every 10
seconds.
You can change the interval via the custom parameter metrics_frequency
.
For example, in order to send Graphite metrics to an endpoint every minute, you
would set the custom parameter metrics_frequency
to 60
using the following command:
cf update-service my-instance -c '{ "metrics_frequency": 60 }'
Metrics Prefix
Depending on your graphite provider, you might need to prefix the metrics with
a certain value, like an API key for example. In this case you can leverage the
custom parameter metrics_prefix
.
cf update-service my-instance -c '{ "metrics_prefix": "my-api-key-for-a-certain-provider" }'
The resulting metric path would have the format:
<metrics_prefix>.<service_guid>.<service_type>.<host>.<metric>
Metrics
These are the all metrics provided by the a9s Elasticsearch.
Metric Id | type | Description |
---|---|---|
metric_fetch_status | Integer (1 = true / 0 = false) | When the last metric update is successful the result is true. |
cluster_health.active_primary_shards | Integer | The number of active primary shards. documentation |
cluster_health.active_shards | Integer | The total number of active primary and replica shards. documentation |
cluster_health.active_shards_percent_as_number | Float | Percent of shards active documentation |
cluster_health.delayed_unassigned_shards | Integer | The number of shards whose allocation has been delayed by this timeout. documentation |
cluster_health.initializing_shards | Integer | The number of shards that are under initialization. documentation |
cluster_health.number_of_data_nodes | Integer | The number of nodes that are dedicated data nodes. documentation |
cluster_health.number_of_in_flight_fetch | Integer | The number of unfinished fetches. documentation |
cluster_health.number_of_nodes | Integer | The number of nodes within the cluster. documentation |
cluster_health.number_of_pending_tasks | Integer | The number of cluster-level changes that have not yet been executed. documentation |
cluster_health.relocating_shards | Integer | The number of shards that are under relocation. documentation |
cluster_health.status | Integer (0 = green / 1 = yellow / 2 = red / 3 = unknown) | Health status of the cluster, based on the state of its primary and replica shards. documentation |
cluster_health.task_max_waiting_in_queue_millis | Integer | The time expressed in milliseconds since the earliest initiated task is waiting for being performed. documentation |
cluster_health.timed_out | Integer (1 = true / 0 = false / 2 = unknown) | If false the response returned within the period of time that is specified by the timeout parameter (30s by default). documentation |
cluster_health.unassigned_shards | Integer | The number of shards that are not allocated documentation |
Examples
*.elasticsearch.*.*.*.*.metric_fetch_status
*.elasticsearch.*.*.*.*.cluster_health.active_primary_shards
*.elasticsearch.*.*.*.*.cluster_health.active_shards
*.elasticsearch.*.*.*.*.cluster_health.active_shards_percent_as_number
*.elasticsearch.*.*.*.*.cluster_health.delayed_unassigned_shards
*.elasticsearch.*.*.*.*.cluster_health.initializing_shards
*.elasticsearch.*.*.*.*.cluster_health.number_of_data_nodes
*.elasticsearch.*.*.*.*.cluster_health.number_of_in_flight_fetch
*.elasticsearch.*.*.*.*.cluster_health.number_of_nodes
*.elasticsearch.*.*.*.*.cluster_health.number_of_pending_tasks
*.elasticsearch.*.*.*.*.cluster_health.relocating_shards
*.elasticsearch.*.*.*.*.cluster_health.status
*.elasticsearch.*.*.*.*.cluster_health.task_max_waiting_in_queue_millis
*.elasticsearch.*.*.*.*.cluster_health.timed_out
*.elasticsearch.*.*.*.*.cluster_health.unassigned_shards
Cloud Foundry Org and Space Guid
The platform operators can enable support on a global level to prefix the Graphite metrics with the CloudFoundry organization and space. This means that all metrics of all service instances (not only yours!) contain that information.
In this case the Graphite metric paths have the following format:
<organization_guid>.<space_guid>.<service_guid>.<service_type>.<host>.<metric>
When you enable in addition the metrics_prefix
for your instance, you will
end up with the metric path format:
<metrics_prefix>.<organization_guid>.<space_guid>.<service_guid>.<service_type>.<host>.<metric>
Add a Syslog Endpoint
The cf update-service
command used with the -c flag can let you stream your syslog to a third-party service. In this case, the command expects a JSON string containing the syslog
key. You can also change the interval for the syslog with the same key used to specify graphite interval
.
cf update-service my-elasticsearch-service \
-c '{ "syslog": ["logs4.your-syslog-endpoint.com:54321"], "interval": "5" }'
Enable Custom Plugins
You can use the cf update-service
command with the -c flag to enable plugins on demand such as the analysis-phonetic plugin which convert tokens to their phonetic representation using Soundex, Metaphone, and a variety of other algorithms.
To enable the analysis phonetic plugin you would use the following command:
cf update-service my-elasticsearch-service \
-c '{"plugins": ["analysis-phonetic"]}'
The currently available plugins are:
repository-s3
(enabled by default, cannot be disabled)repository-azure
(enabled by default, cannot be disabled)analysis-phonetic
Cloud Foundry Application Security Groups
This topic describes how to check whether a security group was created.
Each a9s Data Service will automatically create and update Cloud Foundry security groups in order to protect service instances to be accessed by applications not running in the same Cloud Foundry applications space. To get a better understanding about Security Groups you can have a look on the Understanding Application Security Groups topic.
Get Service Instance GUID
Run cf service INSTANCE_NAME --guid
to get the guid of the service instance.
$ cf service my-elasticsearch --guid
ca16f111-5073-40b7-973a-156c75dd3028
Check available Security Groups
To see all available security groups use cf security-groups
.
$ cf security-groups
Getting security groups as demo@anynines.com
OK
Name Organization Space
#0 public_networks
#1 dns
#2 tcp_open
#3 guard_432fb752-876d-443b-a311-a075f4df2237 demonstrations demo
#4 guard_ca16f111-5073-40b7-973a-156c75dd3028 demonstrations demo
There you can see a security group named guard_ca16f111-5073-40b7-973a-156c75dd3028
was successfully created.
NOTE: in some circumstances the connection between the application and the service instance is not possible, in this case check if a security group was created.
Backup and Restore Service Instances
a9s Elasticsearch provides an easy way to create backups and restore if needed. For a more detailed description, please see the a9s Service Dashboard documentation.
- The backup process is based on Elasticsearch's Snapshot functionality, this means that:
- Backups are not encrypted.
- Due to this, the
backup_manager_encryption_key
property cannot be set.
- Due to this, the
- Backups cannot be downloaded.
- The storage backend is limited to AWS S3 and Azure; generic S3 API services are not supported.
- Backups are not encrypted.
- Elasticsearch's default filter plugins are not available.
Make a Service Instance Locally Available
It is possible to access any of the a9s Data Services locally. That means you can connect with a local client to the service for any purpose such as debbuging. CF provides a smart way to create SSH forward tunnels via a pushed application. For more information about this feature see the Accessing Apps with SSH section of the CF documentation.
First of all you must have an application bound to the service. How to do this see Bind an Application to a Service Instance.
NOTE: cf ssh
support must be enabled in the platform. Ask your administrator if you are not sure.
Get The Service Url and Credentials
When you follow this instructions Obtain Service Instance Access Credentials you will get the hostname of the service and the user credentials.
$ cf env a9s-elasticsearch-app
Getting env variables for app a9s-elasticsearch-app in org test / space test as admin...
OK
System-Provided:
{
"VCAP_SERVICES": {
"a9s-elasticsearch5": [
{
"credentials": {
"host": [
"d67901c.service.dc1.a9svs"
],
"password": "a9s-brk-usr",
"username": "a9s-password"
},
"label": "a9s-elasticsearch5",
"name": "my-elasticsearch-service",
"plan": "elasticsearch-cluster-small"
}
]
}
}
...
Notice the host d67901c.service.dc1.a9svs
, the username a9s-brk-usr
and the password a9s-password
. You will need this in the next step.
Create a Tunnel to The Service
With the cf ssh
as mentioned before you can create a ssh forward tunnel to the management dashboard.
Use port 9200 to connect to the a9s Elasticsearch Instance.
$ cf ssh a9s-elasticsearch-app -L 9200:d67901c.service.dc1.a9svs:9200
vcap@956aaf4e-6da9-4f69-4b1d-8e631a403312:~$
When the ssh tunnel is open you can access the instance over the address localhost:9200
.
NOTE: Don't forget to close the session with exit
.
Setup Disk Usage Alerts
Each service comes with the a9s Parachute. This component monitors ephemeral and persistent disk usage. See the a9s Parachute documentation how to configure the component.
Custom Parameters
As an end user, you have the opportunity to customize your service instance by using custom
parameters. Custom parameters are passed on to a service instance by using the -c
switch of the
cf CLI commands cf create-service
and cf update-service
. For example:
cf update-service my-instance -c '{"ssl_session_timeout": "4h"}'
You don't have to use those settings. There are sane defaults in place that fit your service plan well.
ssl_session_timeout
This property is used to specify a period during which a client may reuse the session parameters
by configuring the setting ssl_session_timeout
of the
nginx sitting in
front of Elasticsearch service instances configured with SSL. The content follows the
nginx configuration file measurement units.