The proper way of configuring syslog services on NSX-T components

This wiki article was written by Iwan Hoogendoorn

The outputs and tests were done by Iwan Hoogendoorn and Sergei Ischenko

Introduction

There is a lot of confusion going on, on how to configure logging on the NSX-T Manager components. I get questions like:

  1. How do we log the Distributed Firewall rules?
  2. What is the source of the logging messages?
  3. How do I see logging for the T0 or T1 Gateways and Gateway firewalls?
  4. Is it possible to configure separate logging servers for different purposes?
  5. Is it possible to log the Distributed Firewall rules to a separate logging server?

In this article, we will try to answer all of these questions with examples.

Topology

Below you will find the topology we are testing with.

Network-Diagram-NSX-T-logging.png

We have used a vRealize Log Insight Server that is not displayed on the drawing to sent our log messages to.

Enable logging on the NSX-T Manager

Before we start we first want to get something out of this world. When you configure the required Syslog commands on the NSX-T Manager the Manager will NOT push the logging servers to the Edge Transport Nodes or on the Transport Nodes. When we configure logging on the manager this will only provide information on what is happening on the manager.

When you have a manager cluster you set up an SSH session to the VIP and you put in these commands:

nsxapp-01a> set logging-server 192.168.110.24 proto udp level info
WARNING - You are configuring udp-based log forwarding. This will send sensitive information unencrypted over the network. The Splunk App for NSX-T only accepts TLS connections.


nsxapp-01a> get logging-servers
192.168.110.24 proto udp level info


When we have put in these commands we can see the following messages showed up in the Logging Server:

Logging-nsxt-210620019-01.png

The SOURCE of this logging messages is the NSX-T Manager itself where the log message is initiated from.

It is also possible to send a certain type of messages to the logging server and this is done by configuring a "messageid".

nsxapp-01a> set logging-server 192.168.110.24 proto udp level info messageid FIREWALL,FIREWALL-PKTLOG
WARNING - You are configuring udp-based log forwarding. This will send sensitive information unencrypted over the network. The Splunk App for NSX-T only accepts TLS connections.


This will only send out a specific type of logging messages to a specific logging server. You can find out more on the about messageid's types here

Enable logging on the Edge Transport Nodes (Edge VM's)

In order to receive logging messages from our T0 and T1 Gateways, we also need to enable logging on the Edge Transport Nodes. We want to stress one more time that the logging server is NOT pushed automatically on the Edge Transport Nodes by the manager.

edgenodi-01a> set logging-server 192.168.110.24 proto udp level info
WARNING - You are configuring udp-based log forwarding. This will send sensitive information unencrypted over the network. The Splunk App for NSX-T only accepts TLS connections.


edgenodi-01a> get logging-servers
192.168.110.24 proto udp level info


We have enabled the Gateway firewall and configured a rule with a tag + logging enabled for testing purposes.

Logging-nsxt-210620019-02.png

Logging-nsxt-210620019-03.png

And we did a ping test from the "outside" world into the "NSX" world:

C:\Users\Administrator>ping 172.16.20.11
 
Pinging 172.16.20.11 with 32 bytes of data:
Reply from 172.16.20.11: bytes=32 time=46ms TTL=61
Reply from 172.16.20.11: bytes=32 time=6ms TTL=61
Reply from 172.16.20.11: bytes=32 time=6ms TTL=61
Reply from 172.16.20.11: bytes=32 time=6ms TTL=61
 
Ping statistics for 172.16.20.11:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 6ms, Maximum = 46ms, Average = 16ms
 
C:\Users\Administrator>


And the logging messages appear nicely in our logging server

Logging-nsxt-210620019-10.png

We have also simulated a BGP peer disruption and this also appeared nicely in our logging server:

Here you see the BGP peer is ESTABLISHED:

edgenode-01a(tier0_sr)> get bgp neighbor summary
BFD States: NC - Not configured, AC - Activating,DC - Disconnected
            AD - Admin down, DW - Down, IN - Init,UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 192.168.240.3  Local AS: 100
 
Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx
 
192.168.240.1                       200         Estab 18:09:18     NC  3373    3423    12     14


Then we bring it down and test it with ping to see if it is really down:

edgenode-01a(tier0_sr)> ping 192.168.240.1
PING 192.168.240.1 (192.168.240.1): 56 data bytes
^C
edgenode-01a(tier0_sr)>
--- 192.168.240.1 ping statistics ---
8 packets transmitted, 0 packets received, 100.0% packet loss


And we verify again, and see the status is now ACTIVE:

edgenode-01a(tier0_sr)> get bgp neighbor summary
BFD States: NC - Not configured, AC - Activating,DC - Disconnected
            AD - Admin down, DW - Down, IN - Init,UP - Up
BGP summary information for VRF default for address-family: ipv4Unicast
Router ID: 192.168.240.3  Local AS: 100
 
Neighbor                            AS          State Up/DownTime  BFD InMsgs  OutMsgs InPfx  OutPfx
 
192.168.240.1                       200         Activ 00:00:26     NC  3373    3427    0      0


And we verify this is the logging server:

Logging-nsxt-210620019-11.png

The SOURCE of this logging messages is the Edge VM itself where the log message is initiated from.

Enable logging on the Host Transport Nodes (ESXi Hosts)

In order to log Distributed Firewall Rules (DFW) you need to enable commands on the Host Transport nodes (ESXi hosts) itself. You don't need any logging configuration on the NSX-T Manager and we tested this by removing the commands on the manager:

nsxapp-01a> del logging-server 192.168.110.24 proto udp level info


nsxapp-01a> get logging-servers
nsxapp-01a>


And we have added the following configuration into the Host Transport nodes:

[root@esxcomp-02a:~] esxcli network firewall ruleset set -r syslog -e true
[root@esxcomp-02a:~] esxcli system syslog config set --loghost=udp://192.168.110.24:514
[root@esxcomp-02a:~] esxcli system syslog reload
[root@esxcomp-02a:~] esxcli system syslog mark -s "This is a test message"


[root@esxcomp-02a:~] esxcli system syslog config get
   Default Network Retry Timeout: 180
   Dropped Log File Rotation Size: 100
   Dropped Log File Rotations: 10
   Enforce SSLCertificates: false
   Local Log Output: /scratch/log
   Local Log Output Is Configured: false
   Local Log Output Is Persistent: true
   Local Logging Default Rotation Size: 1024
   Local Logging Default Rotations: 8
   Log To Unique Subdirectory: false
   Message Queue Drop Mark: 90
   Remote Host: udp://192.168.110.24:514


We have enabled the DFW firewall and configured a rule with a tag + logging enabled for testing purposes.

Logging-nsxt-210620019-08.png

Logging-nsxt-210620019-07.png

And we have done another ping test:

C:\Users\Administrator>ping 172.16.20.11
 
Pinging 172.16.20.11 with 32 bytes of data:
Reply from 172.16.20.11: bytes=32 time=46ms TTL=61
Reply from 172.16.20.11: bytes=32 time=6ms TTL=61
Reply from 172.16.20.11: bytes=32 time=6ms TTL=61
Reply from 172.16.20.11: bytes=32 time=6ms TTL=61
 
Ping statistics for 172.16.20.11:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 6ms, Maximum = 46ms, Average = 16ms
 
C:\Users\Administrator>


And the logging messages appear nicely in our logging server

Logging-nsxt-210620019-09.png

The SOURCE of this logging messages is the Transport Node itself where the log message is initiated from.

It is NOT possible to send the DFW to a separate logging server and split another type of messages and send these to another logging server. If you want this you have two options:

  1. Configure another (second) logging server on the ESXi host so all messages are sent to two different logging servers
  2. Use vRealize Log Insight to send all logging messages to and configure a forwarding with a filter to send out certain logging messages.

Below an example is given on how to configure multiple logging servers on the ESXi host:

[root@esxcomp-02a:~] esxcli system syslog config set --loghost=udp://192.168.110.24:514,udp://192.168.110.111:222


[root@esxcomp-02a:~] esxcli system syslog config get
  Default Network Retry Timeout: 180
  Dropped Log File Rotation Size: 100
  Dropped Log File Rotations: 10
  Enforce SSLCertificates: false
  Local Log Output: /scratch/log
  Local Log Output Is Configured: false
  Local Log Output Is Persistent: true
  Local Logging Default Rotation Size: 1024
  Local Logging Default Rotations: 8
  Log To Unique Subdirectory: false
  Message Queue Drop Mark: 90
  Remote Host: udp://192.168.110.24:514,udp://192.168.110.111:222


And if you want to go for option two then you configure the vRealize Log Insight Server with the below settings

Logging-nsxt-210620019-12.png