Over the past several years of using vRealize Automation 6.x and 7.x, I have generated numerous dashboards and search queries within Splunk to explore the log data generated by these products. Knowing that vRealize Automation 8.0 is an entirely new product compared to previous versions, I decided that it was time to begin reviewing the log data being generated by the appliance to determine what information could be obtained from the logs. This post will discuss the logging capabilities of vRealize Automation 8.0, as well as the data that can be collected by analyzing the logs.
The ability to forward logs to a centralized log server is essential when attempting to operate a private cloud. Knowing that I need to get the log data from the vRealize Automation 8.0 appliance to my Splunk server, I began reading the vRealize Automation 8.0 documentation titled Working with logs in vRealize Automation. The documentation details two methods of obtaining log data from the virtual appliance. The first method provided is the ability to create on-demand log bundles similar to what we’ve been doing for years when opening technical support tickets with VMware. The second method provided is to forward the logs to a remote VMware Log Insight server.
The generation of a log bundle is a simple process withing vRealize Automation 8.0 and is accomplished by utilizing the vracli command. To create a log bundle:
That’s all there is to it. Of course, some additional options and flags can be specified, but for the basic log bundle generation, the above is all you’ll need. If you’re curious about the additional options, you can find further information on the How do I work with logs and log bundles in vRealize Automation documentation page.
Forwarding the vRealize Automation 8.0 logs to a remote server is the option that most people will want to implement. While previous releases of vRealize Automation provided flexible forwarding options for remote log aggregation services, to my surprise, the vRealize Automation 8.0 appliance only supports the forwarding of logs using the VMware Log Insight (vRLI) API via HTTP/HTTPs POSTs.
To configure forwarding of log data to a vRLI deployment:
I found in my lab environment that all of my log entries showed up in vRLI with a source hostname of “symphony-logging-daemonset-vf8md” which matched the runtime name of the Kubernetes pod responsible for aggregating and managing vRealize Automation 8.0’s log data. While some log entries do include the vRA appliance FQDN in a field titled “host”, not all of the log entries include this field. Additionally, all log entries show a source IP address of “127.0.0.1”. Because of the lack of a properly-identified source, determining the source of your vRealize Automation 8.0 logs in vRLI will be quite tricky.
VMware chose to implement remote logging withing vRealize Automation 8.0 using Fluentd. Having never heard of Fluentd, I decided to do a little reading and ran across the blog post titled Forwarding Kubernetes Logs to vRealize Log Insight via Fluentd. From this post, I learned that Fluentd is a popular choice for forwarding logs from Kubernetes environments. It is a powerful solution with many plugins to support multiple log input and output formats. Unfortunately, it appears that vRealize Automation 8.0’s implementation today only supports vRLI as the output type (no Syslog here). We can confirm the supported Fluentd output methods by checking the Fluentd plugins directory within the container. First, we need to determine the Name or ID of the container that runs Fluentd:
[email protected] \[ ~ \]# docker ps | grep symphony 5edaa3595dff 60d42ba9520d "/usr/bin/fluentd --…" 16 minutes ago Up 16 minutes k8s\_symphony-logging-agent\_symphony-logging-daemonset-v2c5t\_prelude\_b4d585ad-3c01-11ea-ab84-0050569671c4\_9 2ffbb2970db5 vmware/pause:3.1 "/pause" 6 hours ago Up 6 hours k8s\_POD\_symphony-logging-daemonset-v2c5t\_prelude\_b4d585ad-3c01-11ea-ab84-0050569671c4\_3
Now that we have the ID of the Docker container executing Fluentd, we need to determine the --plugin option specified when the Fluentd service starts:
[email protected] \[ ~ \]# docker exec -it 5edaa3595dff top Tasks: 5 total, 1 running, 4 sleeping, 0 stopped, 0 zombie Mem: 32945940K total, 25483480K used, 7462460K free, 461496K buffers Swap: 0 total, 0 used, 0 free, 7285928K cached 800%cpu 39%user 0%nice 25%sys 735%idle 0%iow 0%irq 1%sirq 0%host PID USER PR NI VIRT RES SHR S\[%CPU\] %MEM TIME+ ARGS 14 root 20 0 676M 147M 29M S 0.3 0.4 0:06.40 ruby -Eascii-8bit:ascii-8bit /usr/bin/fluentd --config /fluentd/etc/fluent.conf --plugin /fluentd/plugins --under-superv+ 1667 root 20 0 12M 5.8M 3.3M R 0.0 0.0 0:00.00 top 1636 root 20 0 12M 5.8M 3.3M S 0.0 0.0 0:00.00 top 1623 root 20 0 12M 3.8M 3.3M S 0.0 0.0 0:00.00 top 1 root 20 0 90M 71M 10M S 0.0 0.2 0:00.70 ruby /usr/bin/fluentd --config /fluentd/etc/fluent.conf --plugin /fluentd/plugins
Finally, now that we know the path of where the Fluentd plugins are stored, we execute a directory lookup to see the available plugins:
[email protected] \[ ~ \]# docker exec -it 5edaa3595dff ls /fluentd/plugins http\_client.rb out\_loginsight\_http.rb out\_vmware\_log\_intelligence.rb
As you see from the directory listing, only three plugins are available in the appliance: HTTP, Log Insight, and vRealize Log Insight Cloud (formerly VMware Log Intelligence). The lack of Syslog support squashes any plans I have for implementing remote logging of vRealize Automation 8.0 to my Splunk deployment (at least not without a lot of effort to parse the HTTP POST data within Splunk). The only method I see available to me at the moment is to send the logs to vRLI first, and then have vRLI forward the data to my Splunk deployment.
Often when troubleshooting an issue, it can be quite useful to view the logs on the console as a process is loading, or you are attempting an action in the web UI that fails. You can still accomplish this by interacting with Kubernetes. Each of the pods that make up the “prelude” namespace on the appliance can be queried for its logs. First, however, you need to know the name of the pod. The appliance contains the following pods:
abx-service-app, assessment-service-app, automation-ui-app, blueprints-ui-app, catalog-service-app, cgs-service, cgs-ui-app, cmx-service-app, codestream-app, consumer-ui-app, docker-registry, docker-registry-proxy, ebs-app, extensibility-ui-app, form-service-app, identity-service-app, identity-ui-app, migration-service-app, migration-ui-app, nginx-httpd-app, no-license-app, orchestration-ui-app, pgpool, pipeline-ui-app, postgres, project-service-app, provisioning-service-app, provisioning-ui-app, proxy-service, rabbitmq-ha, relocation-service-app, relocation-ui-app, symphony-logging-daemonset, tango-blueprint-service-app, tango-vro-gateway-app, terraform-service-app, user-profile-service-app, vco-app
If you know which service you want to view the logs from, you’ll next need to determine the runtime name of the Kubernetes pod that contains the service. To see the full list of running pods, you execute the following on the command line:
[email protected] \[ / \]# kubectl -n prelude get pods NAME READY STATUS RESTARTS AGE abx-service-app-684468595b-mfcbx 1/1 Running 16 26d assessment-service-app-5d5ff7b6f4-vlm54 1/1 Running 42 26d automation-ui-app-74868745db-7zt8l 1/1 Running 10 26d blueprints-ui-app-6b456495cb-t5qs2 1/1 Running 10 26d catalog-service-app-69678557d5-9snc7 1/1 Running 27 26d cgs-service-b57c4775c-qwr2m 1/1 Running 41 26d cgs-ui-app-7c87b58fb6-t2flr 1/1 Running 10 26d cmx-service-app-867d76dbfd-ph6kn 1/1 Running 13 26d codestream-app-86b5b564bd-8w4h7 1/1 Running 46 26d consumer-ui-app-9cf656775-fnx69 1/1 Running 10 26d docker-registry-5bf9b47b68-glhwc 1/1 Running 13 26d docker-registry-proxy-8cbgh 1/1 Running 46 26d ebs-app-59cd6c994b-shlqp 1/1 Running 15 26d extensibility-ui-app-6669df67cb-szdkb 1/1 Running 10 26d form-service-app-5ccff5ff57-47l46 1/1 Running 41 26d identity-service-app-759c68c8d-h68gt 1/1 Running 55 26d identity-ui-app-558768574-2dnhq 1/1 Running 10 26d migration-service-app-bb4c86989-ff55g 1/1 Running 41 26d migration-ui-app-74cbb89796-vmvs7 1/1 Running 10 26d nginx-httpd-app-6c7944648b-798jc 1/1 Running 10 26d no-license-app-s4htq 1/1 Running 10 26d orchestration-ui-app-75c9fc6ccb-zx45f 1/1 Running 10 26d pgpool-fd68cb474-smjmf 1/1 Running 37 26d pipeline-ui-app-f57c659cd-cf759 1/1 Running 10 26d postgres-0 1/1 Running 10 26d project-service-app-7477578845-lcskd 1/1 Running 14 26d provisioning-service-app-787856d447-sfczl 1/1 Running 13 26d provisioning-ui-app-9dc7dcd9d-vwlh5 1/1 Running 10 26d proxy-service-tb6lv 1/1 Running 10 26d rabbitmq-ha-0 1/1 Running 10 26d relocation-service-app-5bcd574d45-98xp4 1/1 Running 30 26d relocation-ui-app-84697cb968-cb9vv 1/1 Running 10 26d symphony-logging-daemonset-v2c5t 1/1 Running 13 14d tango-blueprint-service-app-8b7f75f7d-rgt6n 1/1 Running 34 26d tango-vro-gateway-app-59ff6cddfd-bvwrm 1/1 Running 34 26d terraform-service-app-76757767cc-cxvb4 2/2 Running 20 26d user-profile-service-app-74d6598975-s9xkz 1/1 Running 49 26d vco-app-5857fdc55f-dmh8k 2/2 Running 18 26d
Now that you have the runtime names for all of the pods, you’ll query the logs from the “rabbitmq-ha-0” pod.
[email protected] \[ / \]# kubectl logs -n prelude rabbitmq-ha-0 total 20 drwxrwxrwx 5 root root 4096 Feb 4 13:10 . drwxr-xr-x 1 root root 4096 Nov 27 14:40 .. drwxrwxrwx 3 rabbitmq rabbitmq 4096 Feb 1 20:55 config drwxrwxrwx 4 rabbitmq rabbitmq 4096 Feb 1 20:55 mnesia drwxrwxrwx 2 rabbitmq rabbitmq 4096 Dec 31 17:24 schema 2020-02-04 13:10:37.793 \[info\] <0.8.0> Feature flags: list of feature flags found: 2020-02-04 13:10:37.793 \[info\] <0.8.0> Feature flags: feature flag states written to disk: yes 2020-02-04 13:10:37.893 \[info\] <0.256.0> Starting RabbitMQ 3.7.17 on Erlang 22.0.7 Copyright (C) 2007-2019 Pivotal Software, Inc. Licensed under the MPL. See https://www.rabbitmq.com/ ## ## ## ## RabbitMQ 3.7.17. Copyright (C) 2007-2019 Pivotal Software, Inc. ########## Licensed under the MPL. See https://www.rabbitmq.com/ ###### ## ########## Logs: <stdout> Starting broker... 2020-02-04 13:10:37.893 \[info\] <0.256.0> node : rabbi[email protected] home dir : /var/lib/rabbitmq config file(s) : /etc/rabbitmq/rabbitmq.conf cookie hash : GVrUAEuiF7v1LP7aGSh7kw== log(s) : <stdout> database dir : /var/lib/rabbitmq/mnesia/rabbi[email protected] 2020-02-04 13:10:37.906 \[info\] <0.256.0> Running boot step pre\_boot defined by app rabbit
To follow the logs as they are generated, add a “-f” to the command above (kubectl logs -f -n prelude rabbitmq-ha-0). As you can see, it’s easy to view the logs from the service.
Lack of support for basic Syslog functionality within an enterprise cloud management platform seems like a significant oversight and a complete step backward from vRealize Automation 7.x. Additionally, the logging functionality that does exist via integration with vRealize Log Insight appears to be half baked at best as the source of the log data is shown as the Kubernetes pod name, which changes each time the pods restart. Hopefully, VMware will focus some effort on refining the logging functionality in future updates of vRealize Automation 8.x to provide at a minimum the same level of functionality as they supported in vRealize Automation 7.x.
Get Notified of Future Posts