VMware vRealize Automation 8.0 Logs

January 28, 2020

Reading time: 8 minutes

Over the past several years of using vRealize Automation 6.x and 7.x, I have generated numerous dashboards and search queries within Splunk to explore the log data generated by these products. Knowing that vRealize Automation 8.0 is an entirely new product compared to previous versions, I decided that it was time to begin reviewing the log data being generated by the appliance to determine what information could be obtained from the logs. This post will discuss the logging capabilities of vRealize Automation 8.0, as well as the data that can be collected by analyzing the logs.

How to Access the Logs

The ability to forward logs to a centralized log server is essential when attempting to operate a private cloud. Knowing that I need to get the log data from the vRealize Automation 8.0 appliance to my Splunk server, I began reading the vRealize Automation 8.0 documentation titled Working with logs in vRealize Automation. The documentation details two methods of obtaining log data from the virtual appliance. The first method provided is the ability to create on-demand log bundles similar to what we’ve been doing for years when opening technical support tickets with VMware. The second method provided is to forward the logs to a remote VMware Log Insight server.

Generating a Log Bundle

The generation of a log bundle is a simple process withing vRealize Automation 8.0 and is accomplished by utilizing the vracli command. To create a log bundle:

SSH to the vRealize Automation appliance
Issue the command vracli log-bundle
(if you are running a clustered deployment, you only need to issue the command from one appliance)
Copy the log bundle from the path where you executed the command

That’s all there is to it. Of course, some additional options and flags can be specified, but for the basic log bundle generation, the above is all you’ll need. If you’re curious about the additional options, you can find further information on the How do I work with logs and log bundles in vRealize Automation documentation page.

Forwarding Logs to a Remote Server

Forwarding the vRealize Automation 8.0 logs to a remote server is the option that most people will want to implement. While previous releases of vRealize Automation provided flexible forwarding options for remote log aggregation services, to my surprise, the vRealize Automation 8.0 appliance only supports the forwarding of logs using the VMware Log Insight (vRLI) API via HTTP/HTTPs POSTs.

To configure forwarding of log data to a vRLI deployment:

SSH to the vRealize Automation appliance
Issue the command vracli vrli {vRLI FQDN}

I found in my lab environment that all of my log entries showed up in vRLI with a source hostname of “symphony-logging-daemonset-vf8md” which matched the runtime name of the Kubernetes pod responsible for aggregating and managing vRealize Automation 8.0’s log data. While some log entries do include the vRA appliance FQDN in a field titled “host”, not all of the log entries include this field. Additionally, all log entries show a source IP address of “127.0.0.1”. Because of the lack of a properly-identified source, determining the source of your vRealize Automation 8.0 logs in vRLI will be quite tricky.

What About Syslog?

VMware chose to implement remote logging withing vRealize Automation 8.0 using Fluentd. Having never heard of Fluentd, I decided to do a little reading and ran across the blog post titled Forwarding Kubernetes Logs to vRealize Log Insight via Fluentd. From this post, I learned that Fluentd is a popular choice for forwarding logs from Kubernetes environments. It is a powerful solution with many plugins to support multiple log input and output formats. Unfortunately, it appears that vRealize Automation 8.0’s implementation today only supports vRLI as the output type (no Syslog here). We can confirm the supported Fluentd output methods by checking the Fluentd plugins directory within the container. First, we need to determine the Name or ID of the container that runs Fluentd:

root@vlab-vra \[ ~ \]# docker ps | grep symphony
5edaa3595dff        60d42ba9520d        "/usr/bin/fluentd --…"   16 minutes ago      Up 16 minutes                           k8s\_symphony-logging-agent\_symphony-logging-daemonset-v2c5t\_prelude\_b4d585ad-3c01-11ea-ab84-0050569671c4\_9
2ffbb2970db5        vmware/pause:3.1    "/pause"                 6 hours ago         Up 6 hours                              k8s\_POD\_symphony-logging-daemonset-v2c5t\_prelude\_b4d585ad-3c01-11ea-ab84-0050569671c4\_3

Now that we have the ID of the Docker container executing Fluentd, we need to determine the --plugin option specified when the Fluentd service starts:

root@vlab-vra \[ ~ \]# docker exec -it 5edaa3595dff top 

Tasks: 5 total,   1 running,   4 sleeping,   0 stopped,   0 zombie
  Mem: 32945940K total, 25483480K used,  7462460K free,   461496K buffers
 Swap:         0 total,         0 used,         0 free,  7285928K cached
800%cpu  39%user   0%nice  25%sys 735%idle   0%iow   0%irq   1%sirq   0%host
  PID USER         PR  NI VIRT  RES  SHR S\[%CPU\] %MEM     TIME+ ARGS
   14 root         20   0 676M 147M  29M S  0.3   0.4   0:06.40 ruby -Eascii-8bit:ascii-8bit /usr/bin/fluentd --config /fluentd/etc/fluent.conf --plugin /fluentd/plugins --under-superv+
 1667 root         20   0  12M 5.8M 3.3M R  0.0   0.0   0:00.00 top
 1636 root         20   0  12M 5.8M 3.3M S  0.0   0.0   0:00.00 top
 1623 root         20   0  12M 3.8M 3.3M S  0.0   0.0   0:00.00 top
    1 root         20   0  90M  71M  10M S  0.0   0.2   0:00.70 ruby /usr/bin/fluentd --config /fluentd/etc/fluent.conf --plugin /fluentd/plugins

Finally, now that we know the path of where the Fluentd plugins are stored, we execute a directory lookup to see the available plugins:

root@vlab-vra \[ ~ \]# docker exec -it 5edaa3595dff ls /fluentd/plugins
http\_client.rb  out\_loginsight\_http.rb  out\_vmware\_log\_intelligence.rb

As you see from the directory listing, only three plugins are available in the appliance: HTTP, Log Insight, and vRealize Log Insight Cloud (formerly VMware Log Intelligence). The lack of Syslog support squashes any plans I have for implementing remote logging of vRealize Automation 8.0 to my Splunk deployment (at least not without a lot of effort to parse the HTTP POST data within Splunk). The only method I see available to me at the moment is to send the logs to vRLI first, and then have vRLI forward the data to my Splunk deployment.

Access Logs from the CLI

Often when troubleshooting an issue, it can be quite useful to view the logs on the console as a process is loading, or you are attempting an action in the web UI that fails. You can still accomplish this by interacting with Kubernetes. Each of the pods that make up the “prelude” namespace on the appliance can be queried for its logs. First, however, you need to know the name of the pod. The appliance contains the following pods:

abx-service-app, assessment-service-app, automation-ui-app, blueprints-ui-app, catalog-service-app, cgs-service, cgs-ui-app, cmx-service-app, codestream-app, consumer-ui-app, docker-registry, docker-registry-proxy, ebs-app, extensibility-ui-app, form-service-app, identity-service-app, identity-ui-app, migration-service-app, migration-ui-app, nginx-httpd-app, no-license-app, orchestration-ui-app, pgpool, pipeline-ui-app, postgres, project-service-app, provisioning-service-app, provisioning-ui-app, proxy-service, rabbitmq-ha, relocation-service-app, relocation-ui-app, symphony-logging-daemonset, tango-blueprint-service-app, tango-vro-gateway-app, terraform-service-app, user-profile-service-app, vco-app

If you know which service you want to view the logs from, you’ll next need to determine the runtime name of the Kubernetes pod that contains the service. To see the full list of running pods, you execute the following on the command line:

root@vlab-vra \[ / \]# kubectl -n prelude get pods
NAME                                          READY   STATUS    RESTARTS   AGE
abx-service-app-684468595b-mfcbx              1/1     Running   16         26d
assessment-service-app-5d5ff7b6f4-vlm54       1/1     Running   42         26d
automation-ui-app-74868745db-7zt8l            1/1     Running   10         26d
blueprints-ui-app-6b456495cb-t5qs2            1/1     Running   10         26d
catalog-service-app-69678557d5-9snc7          1/1     Running   27         26d
cgs-service-b57c4775c-qwr2m                   1/1     Running   41         26d
cgs-ui-app-7c87b58fb6-t2flr                   1/1     Running   10         26d
cmx-service-app-867d76dbfd-ph6kn              1/1     Running   13         26d
codestream-app-86b5b564bd-8w4h7               1/1     Running   46         26d
consumer-ui-app-9cf656775-fnx69               1/1     Running   10         26d
docker-registry-5bf9b47b68-glhwc              1/1     Running   13         26d
docker-registry-proxy-8cbgh                   1/1     Running   46         26d
ebs-app-59cd6c994b-shlqp                      1/1     Running   15         26d
extensibility-ui-app-6669df67cb-szdkb         1/1     Running   10         26d
form-service-app-5ccff5ff57-47l46             1/1     Running   41         26d
identity-service-app-759c68c8d-h68gt          1/1     Running   55         26d
identity-ui-app-558768574-2dnhq               1/1     Running   10         26d
migration-service-app-bb4c86989-ff55g         1/1     Running   41         26d
migration-ui-app-74cbb89796-vmvs7             1/1     Running   10         26d
nginx-httpd-app-6c7944648b-798jc              1/1     Running   10         26d
no-license-app-s4htq                          1/1     Running   10         26d
orchestration-ui-app-75c9fc6ccb-zx45f         1/1     Running   10         26d
pgpool-fd68cb474-smjmf                        1/1     Running   37         26d
pipeline-ui-app-f57c659cd-cf759               1/1     Running   10         26d
postgres-0                                    1/1     Running   10         26d
project-service-app-7477578845-lcskd          1/1     Running   14         26d
provisioning-service-app-787856d447-sfczl     1/1     Running   13         26d
provisioning-ui-app-9dc7dcd9d-vwlh5           1/1     Running   10         26d
proxy-service-tb6lv                           1/1     Running   10         26d
rabbitmq-ha-0                                 1/1     Running   10         26d
relocation-service-app-5bcd574d45-98xp4       1/1     Running   30         26d
relocation-ui-app-84697cb968-cb9vv            1/1     Running   10         26d
symphony-logging-daemonset-v2c5t              1/1     Running   13         14d
tango-blueprint-service-app-8b7f75f7d-rgt6n   1/1     Running   34         26d
tango-vro-gateway-app-59ff6cddfd-bvwrm        1/1     Running   34         26d
terraform-service-app-76757767cc-cxvb4        2/2     Running   20         26d
user-profile-service-app-74d6598975-s9xkz     1/1     Running   49         26d
vco-app-5857fdc55f-dmh8k                      2/2     Running   18         26d

Now that you have the runtime names for all of the pods, you’ll query the logs from the “rabbitmq-ha-0” pod.

root@vlab-vra \[ / \]# kubectl logs -n prelude rabbitmq-ha-0
total 20
drwxrwxrwx 5 root     root     4096 Feb  4 13:10 .
drwxr-xr-x 1 root     root     4096 Nov 27 14:40 ..
drwxrwxrwx 3 rabbitmq rabbitmq 4096 Feb  1 20:55 config
drwxrwxrwx 4 rabbitmq rabbitmq 4096 Feb  1 20:55 mnesia
drwxrwxrwx 2 rabbitmq rabbitmq 4096 Dec 31 17:24 schema
2020-02-04 13:10:37.793 \[info\] <0.8.0> Feature flags: list of feature flags found:
2020-02-04 13:10:37.793 \[info\] <0.8.0> Feature flags: feature flag states written to disk: yes
2020-02-04 13:10:37.893 \[info\] <0.256.0>
 Starting RabbitMQ 3.7.17 on Erlang 22.0.7
 Copyright (C) 2007-2019 Pivotal Software, Inc.
 Licensed under the MPL.  See https://www.rabbitmq.com/

  ##  ##
  ##  ##      RabbitMQ 3.7.17. Copyright (C) 2007-2019 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See https://www.rabbitmq.com/
  ######  ##
  ##########  Logs: <stdout>

              Starting broker...
2020-02-04 13:10:37.893 \[info\] <0.256.0>
 node           : rabbit@rabbitmq-ha-0.rabbitmq-ha-discovery.prelude.svc.cluster.local
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.conf
 cookie hash    : GVrUAEuiF7v1LP7aGSh7kw==
 log(s)         : <stdout>
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-ha-0.rabbitmq-ha-discovery.prelude.svc.cluster.local
2020-02-04 13:10:37.906 \[info\] <0.256.0> Running boot step pre\_boot defined by app rabbit

To follow the logs as they are generated, add a “-f” to the command above (kubectl logs -f -n prelude rabbitmq-ha-0). As you can see, it’s easy to view the logs from the service.

Final Thoughts

Lack of support for basic Syslog functionality within an enterprise cloud management platform seems like a significant oversight and a complete step backward from vRealize Automation 7.x. Additionally, the logging functionality that does exist via integration with vRealize Log Insight appears to be half baked at best as the source of the log data is shown as the Kubernetes pod name, which changes each time the pods restart. Hopefully, VMware will focus some effort on refining the logging functionality in future updates of vRealize Automation 8.x to provide at a minimum the same level of functionality as they supported in vRealize Automation 7.x.

Tags:

Troubleshooting