VM packet tracing from NSX Manager CLI

TL;DR – The content of this post is also covered in the following YouTube video: https://goo.gl/w9hx1r, which shows how to do a packet trace of a VM running in NSX, how to export it into an admin desktop by simply entering a few commands on the NSX Manager CLI, and how to analyze it with Wireshark.

As you may already know, NSX is VMware’s solution for deploying network and security policies in just a few seconds without reconfiguring any physical device.

It includes many monitoring and visibility tools that dramatically simplify day to day operations, like vRealize Network Insight, vRealize Log Insight, NSX Trace Flow or NSX Flow Monitoring. If interested, you can view them in action in the demo “NSX Series 05 – Monitoring and Visibility” in my YouTube channel youtube.com/AVillarGarea

screen-shot-2016-11-22-at-17-09-18

 

That said, this article focuses in one of the improvements to NSX Central CLI introduced in NSX 6.2.3, that allows to take and export a trace from any VM running on NSX by simply entering a few commands on the NSX Manager CLI. This is very useful when an admin needs to carefully analyze the capture by, for example, opening it with Wireshark on his/hers desktop.

In order to take the trace, we first need to identify the following three parameters:

  • host-id – the host where the VM resides
  • vm-id – the VM we want to get the trace from
  • vnic-id – of all possible VM vnics, the one we want to trace

The great thing of NSX Central CLI is that we can get all the required info and the trace itself from the same place, the NSX Manager CLI. So let’s ssh into the NSX manager!!

The first task is to identify the VM vnic-id. We start by listing all available clusters and then we will drill down to find the VM we want to trace:

nsxmgr-01a> show cluster all
No.  Cluster Name       Cluster Id               Datacenter Name   Firewall Status
1    RegionA01-MGMT01   domain-c71               RegionA01         Enabled
2    RegionA01-COMP02   domain-c161              RegionA01         Enabled
3    RegionA01-COMP01   domain-c26               RegionA01         Enabled
nsxmgr-01a>

 

In our example, we will take the trace from VM web-11, which sits on esx-01a on RegionA01-COMP01 cluster, so the next step is to get the details of such cluster. The command requires the Cluster-Id, not the cluster name:

nsxmgr-01a> show cluster domain-c26
Datacenter: RegionA01
Cluster: RegionA01-COMP01
No.  Host Name            Host Id                  Installation Status
1    esx-02a.corp.local   host-31                  Enabled
2    esx-01a.corp.local   host-29                  Enabled
nsxmgr-01a>

 

Next we enter a similar command to get the list of VMs on the host, one of which is the one we want to trace (web-11):

nsxmgr-01a> show host host-29
Datacenter: RegionA01
Cluster: RegionA01-COMP01
Host: esx-01a.corp.local
No.  VM Name          VM Id     Power Status
1    private-11       vm-472    on
2    l2vpn-11         vm-477    on
3    web-11           vm-464    on
4    L2VPN-Server-0   vm-475    on
5    app-11           vm-468    on
nsxmgr-01a>

 

Finally, we get the details of our VM, including its vnic-id:

nsxmgr-01a> show vm vm-464
Datacenter: RegionA01
Cluster: RegionA01-COMP01
Host: esx-01a.corp.local
Host-ID: host-29
VM: web-11
Virtual Nics List:
1.
Vnic Name      web-11 - Network adapter 1
Vnic Id        501c7db8-e74f-3aac-86ef-99347b1b95a7.000
Filters        nic-5904752-eth0-vmware-sfw.2
nsxmgr-01a>

 

Once we have identified host-id and vnic-id we can move to do the actual trace. Packet Capture commands are only available under the privileged mode of the NSX Manager CLI, so we first need to enter the enable command and corresponding password to gain the right access:

nsxmgr-01a>
nsxmgr-01a> enable
Password:
nsxmgr-01a#
nsxmgr-01a#

 

The command to start the trace requires identifying the direction of the capture as it would be seen from the network, not from the VM:

  • input – packets entering the network, i.e., packets that have the VM we are monitoring as its source
  • output – packets leaving the network, i.e., packets that have the VM we are monitoring as its destination

In this example, we are taking both traces simultaneously, so we can later combine them and have the full picture of what’s going on with our VM.

Packet capture commands also allow the use tcpdump filters (parameters) for capturing only specific packets. In this case, we are tracing everything so initially we won’t use them.

Considering all the above, we enter the command debug packet capture twice, once to start the input trace and a second time to start the output trace. Remember we need to specify the host-id and the vnic-id:

nsxmgr-01a# debug packet capture host host-29 vnic 501c7db8-e74f-3aac-86ef-99347b1b95a7.000 dir input parameters
Session: fb39d938-0602-486f-8f3f-c4987f9a06ef
Request:
        Capture host: host-29
        Vnic: 501c7db8-e74f-3aac-86ef-99347b1b95a7.000
        Capture point: vnic
        Capture direction: input
Session file: /tmp/pktcap/fb39d938-0602-486f-8f3f-c4987f9a06ef.pcap
Session status: started
nsxmgr-01a#
nsxmgr-01a#
nsxmgr-01a# debug packet capture host host-29 vnic 501c7db8-e74f-3aac-86ef-99347b1b95a7.000 dir output parameters
Session: 325fede9-b2c9-4987-a46d-79d4cfedc292
Request:
        Capture host: host-29
        Vnic: 501c7db8-e74f-3aac-86ef-99347b1b95a7.000
        Capture point: vnic
        Capture direction: output
Session file: /tmp/pktcap/325fede9-b2c9-4987-a46d-79d4cfedc292.pcap
Session status: started
nsxmgr-01a#

 

NSX Manager returns details of each capture, including their session-ids, the direction of the captures and on which vnic and host they are being taken.

Once time enough has passed for the captures to have significant information, we can stop them. For that, we simply enter the no debug packet capture command followed by the session-id we want to stop:

nsxmgr-01a# no debug packet capture session fb39d938-0602-486f-8f3f-c4987f9a06ef
Session: fb39d938-0602-486f-8f3f-c4987f9a06ef
Request:
        Capture host: host-29
        Vnic: 501c7db8-e74f-3aac-86ef-99347b1b95a7.000
        Capture point: vnic
        Capture direction: input
Session file: /tmp/pktcap/fb39d938-0602-486f-8f3f-c4987f9a06ef.pcap
Session status: stopped
nsxmgr-01a#
nsxmgr-01a#
nsxmgr-01a# no debug packet capture session 325fede9-b2c9-4987-a46d-79d4cfedc292
Session: 325fede9-b2c9-4987-a46d-79d4cfedc292
Request:
        Capture host: host-29
        Vnic: 501c7db8-e74f-3aac-86ef-99347b1b95a7.000
        Capture point: vnic
        Capture direction: output
Session file: /tmp/pktcap/325fede9-b2c9-4987-a46d-79d4cfedc292.pcap
Session status: stopped
nsxmgr-01a#

 

Note that in both cases the session status has changed from started to stopped. Once the captures are stopped, we can view them in the NSX Manager CLI (we will see later how to export them). First we will check the full capture details with the command debug packet capture display:

nsxmgr-01a# debug packet capture display session fb39d938-0602-486f-8f3f-c4987f9a06ef parameters
Session: fb39d938-0602-486f-8f3f-c4987f9a06ef
Request:
        Capture host: host-29
        Vnic: 501c7db8-e74f-3aac-86ef-99347b1b95a7.000
        Capture point: vnic
        Capture direction: input
Session file: /tmp/pktcap/fb39d938-0602-486f-8f3f-c4987f9a06ef.pcap
Session status: finished
Capture packets:
reading from file /tmp/pktcap/fb39d938-0602-486f-8f3f-c4987f9a06ef.pcap, link-type EN10MB (Ethernet)
18:23:18.109453 ARP, Reply 172.16.10.11 is-at 00:50:56:9c:4c:9a (oui Unknown), length 46
18:23:18.985453 IP 172.16.10.11 > 172.16.20.21: ICMP echo reply, id 8457, seq 1, length 64
18:23:19.102159 IP 172.16.10.11 > 172.16.20.21: ICMP echo reply, id 8457, seq 2, length 64
18:23:20.103346 IP 172.16.10.11 > 172.16.20.21: ICMP echo reply, id 8457, seq 3, length 64
18:23:21.105233 IP 172.16.10.11 > 172.16.20.21: ICMP echo reply, id 8457, seq 4, length 64
18:23:22.107797 IP 172.16.10.11 > 172.16.20.21: ICMP echo reply, id 8457, seq 5, length 64
18:23:23.107189 IP 172.16.10.11 > 172.16.20.21: ICMP echo reply, id 8457, seq 6, length 64
18:23:24.000731 ARP, Request who-has 172.16.10.1 tell 172.16.10.11, length 46
18:23:24.108074 IP 172.16.10.11 > 172.16.20.21: ICMP echo reply, id 8457, seq 7, length 64
18:23:29.384445 IP 172.16.10.11.http > 172.16.20.21.35396: Flags [S.], seq 587005712, ack 3351899227, win 28960, options [mss 1460,sackOK,TS val 48910469 ecr 48799286,nop,wscale 7], length 0
18:23:29.398822 IP 172.16.10.11.http > 172.16.20.21.35396: Flags [.], ack 401, win 235, options [nop,nop,TS val 48910473 ecr 48799290], length 0
18:23:29.417377 IP 172.16.10.11.http > 172.16.20.21.35396: Flags [P.], seq 1:783, ack 401, win 235, options [nop,nop,TS val 48910478 ecr 48799290], length 782: HTTP: HTTP/1.1 200 OK
18:23:30.117835 IP 172.16.10.11.http > 172.16.20.21.35396: Flags [P.], seq 783:1564, ack 801, win 243, options [nop,nop,TS val 48910653 ecr 48799469], length 781: HTTP: HTTP/1.1 200 OK
18:23:30.794070 IP 172.16.10.11.http > 172.16.20.21.35396: Flags [P.], seq 1564:2345, ack 1201, win 252, options [nop,nop,TS val 48910822 ecr 48799638], length 781: HTTP: HTTP/1.1 200 OK
18:23:31.516909 IP 172.16.10.11.http > 172.16.20.21.35396: Flags [P.], seq 2345:3126, ack 1601, win 260, options [nop,nop,TS val 48911003 ecr 48799818], length 781: HTTP: HTTP/1.1 200 OK
18:23:32.135197 IP 172.16.10.11.http > 172.16.20.21.35396: Flags [P.], seq 3126:3907, ack 2001, win 269, options [nop,nop,TS val 48911157 ecr 48799973], length 781: HTTP: HTTP/1.1 200 OK
18:23:32.421326 IP 172.16.10.11.http > 172.16.20.21.35396: Flags [P.], seq 3907:4688, ack 2401, win 277, options [nop,nop,TS val 48911229 ecr 48800044], length 781: HTTP: HTTP/1.1 200 OK
18:23:37.430787 IP 172.16.10.11.http > 172.16.20.21.35396: Flags [F.], seq 4688, ack 2401, win 277, options [nop,nop,TS val 48912481 ecr 48800046], length 0
18:23:40.959668 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [S.], seq 3811097066, ack 3744778935, win 28960, options [mss 1460,sackOK,TS val 48913363 ecr 48802180,nop,wscale 7], length 0
18:23:40.974549 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [.], ack 44, win 227, options [nop,nop,TS val 48913367 ecr 48802184], length 0
18:23:40.993559 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [P.], seq 1:44, ack 44, win 227, options [nop,nop,TS val 48913372 ecr 48802184], length 43
18:23:40.997585 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [.], seq 44:1492, ack 44, win 227, options [nop,nop,TS val 48913373 ecr 48802189], length 1448
18:23:40.998592 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [P.], seq 1492:1692, ack 44, win 227, options [nop,nop,TS val 48913373 ecr 48802189], length 200
18:23:40.999657 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [.], ack 2012, win 272, options [nop,nop,TS val 48913373 ecr 48802190], length
nsxmgr-01a#

 

We are seeing the input trace, so remember the source IP of the packets belongs to the VM we are monitoring, as previously mentioned.

Next we filter the output using the parameters option. In this case, we repeat the same command adding the keywords port 22 at the end, so that only SSH packets are displayed:

nsxmgr-01a# debug packet capture display session fb39d938-0602-486f-8f3f-c4987f9a06ef parameters port 22
Session: fb39d938-0602-486f-8f3f-c4987f9a06ef
Request:
        Capture host: host-29
        Vnic: 501c7db8-e74f-3aac-86ef-99347b1b95a7.000
        Capture point: vnic
        Capture direction: input
Session file: /tmp/pktcap/fb39d938-0602-486f-8f3f-c4987f9a06ef.pcap
Session status: finished
Capture packets:
reading from file /tmp/pktcap/fb39d938-0602-486f-8f3f-c4987f9a06ef.pcap, link-type EN10MB (Ethernet)
18:23:40.959668 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [S.], seq 3811097066, ack 3744778935, win 28960, options [mss 1460,sackOK,TS val 48913363 ecr 48802180,nop,wscale 7], length 0
18:23:40.974549 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [.], ack 44, win 227, options [nop,nop,TS val 48913367 ecr 48802184], length 0
18:23:40.993559 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [P.], seq 1:44, ack 44, win 227, options [nop,nop,TS val 48913372 ecr 48802184], length 43
18:23:40.997585 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [.], seq 44:1492, ack 44, win 227, options [nop,nop,TS val 48913373 ecr 48802189], length 1448
18:23:40.998592 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [P.], seq 1492:1692, ack 44, win 227, options [nop,nop,TS val 48913373 ecr 48802189], length 200
18:23:40.999657 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [.], ack 2012, win 272, options [nop,nop,TS val 48913373 ecr 48802190], length 0
18:23:41.020137 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [P.], seq 1692:1972, ack 2060, win 272, options [nop,nop,TS val 48913378 ecr 48802193], length 280
18:23:41.072476 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [.], ack 2076, win 272, options [nop,nop,TS val 48913392 ecr 48802199], length 0
18:23:41.074995 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [.], ack 2128, win 272, options [nop,nop,TS val 48913392 ecr 48802209], length 0
18:23:41.076158 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [P.], seq 1972:2024, ack 2128, win 272, options [nop,nop,TS val 48913392 ecr 48802209], length 52
18:23:41.084064 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [P.], seq 2024:2076, ack 2212, win 272, options [nop,nop,TS val 48913394 ecr 48802210], length 52
18:23:43.952422 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [.], ack 2360, win 295, options [nop,nop,TS val 48914112 ecr 48802919], length 0
18:23:45.926724 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [P.], seq 2076:2128, ack 2360, win 295, options [nop,nop,TS val 48914605 ecr 48802919], length 52
18:23:49.708457 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [.], ack 2508, win 317, options [nop,nop,TS val 48915550 ecr 48804367], length 0
18:23:49.727385 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [P.], seq 2128:2164, ack 2508, win 317, options [nop,nop,TS val 48915555 ecr 48804367], length 36
18:23:49.768821 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [.], ack 2628, win 317, options [nop,nop,TS val 48915566 ecr 48804373], length 0
18:23:50.361388 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [P.], seq 2164:2216, ack 2628, win 317, options [nop,nop,TS val 48915714 ecr 48804373], length 52
18:23:50.363923 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [.], ack 3748, win 340, options [nop,nop,TS val 48915714 ecr 48804531], length 0
18:23:50.371543 IP 172.16.10.11.ssh > 172.16.20.21.59434: Flags [P.], seq 2216:2324, ack 3748, win 340, options [nop,nop,TS val 48915716 ecr 48804531], length 108
byte 3243
nsxmgr-01a#

 

Been able to see the trace on the NSX Manager CLI itself can be useful, but sometimes it is still required to export the trace for further analysis with Wireshark or other tools. When exporting, the NSX Manager establishes an SCP session to an SCP server, so the admin desktop must be equipped with one. There are several free options available on the Internet.

We will need the session IDs of both packet captures in order to export them. We can get these IDs at any time by entering the command show packet capture sessions. The final s of sessions is really important, otherwise the system will return a command incomplete warning:

nsxmgr-01a# show packet capture sessions
Sessions:
Session: fb39d938-0602-486f-8f3f-c4987f9a06ef
Request:
        Capture host: host-29
        Vnic: 501c7db8-e74f-3aac-86ef-99347b1b95a7.000
        Capture point: vnic
        Capture direction: input
Session file: /tmp/pktcap/fb39d938-0602-486f-8f3f-c4987f9a06ef.pcap
Session status: finished

Session: 325fede9-b2c9-4987-a46d-79d4cfedc292
Request:
        Capture host: host-29
        Vnic: 501c7db8-e74f-3aac-86ef-99347b1b95a7.000
        Capture point: vnic
        Capture direction: output
Session file: /tmp/pktcap/325fede9-b2c9-4987-a46d-79d4cfedc292.pcap
Session status: finished

======================================
Started session: 0
Stopped session: 0
Finished session: 2
Error session: 0
nsxmgr-01a#

 

Once the SCP server is set and we have the session ids by hand, we can proceed with the copy, for which we use the command debug packet capture scp as shown below:

nsxmgr-01a# debug packet capture scp session fb39d938-0602-486f-8f3f-c4987f9a06ef url admin@192.168.110.10:/
Session: fb39d938-0602-486f-8f3f-c4987f9a06ef
Request:
        Capture host: host-29
        Vnic: 501c7db8-e74f-3aac-86ef-99347b1b95a7.000
        Capture point: vnic
        Capture direction: input
Session file: /tmp/pktcap/fb39d938-0602-486f-8f3f-c4987f9a06ef.pcap
Session status: finished
Begin SCP:
Could not create directory '/var/empty/.ssh'.
The authenticity of host '192.168.110.10 (192.168.110.10)' can't be established.
RSA key fingerprint is SHA256:I9aPF/MDq6ImYqe1ScH2uiyNVcJNNcT4Uu+Hg99SxzI.
Are you sure you want to continue connecting (yes/no)?
Failed to add the host to the list of known hosts (/var/empty/.ssh/known_hosts).
admin@192.168.110.10's password:
nsxmgr-01a#

 

The parameters admin@192.168.110.10:/ specify the following:

  • admin – existing user ID on the SCP server
  • 192.168.110.10 – IP address of the SCP server, in this case the admin desktop
  • : / – path to the folder where the traces will be stored. In the example, the SCP server root folder

At this point the first capture is already available in .pcap format on the admin desktop. Let’s repeat the command to export the second session as well:

nsxmgr-01a# debug packet capture scp session 325fede9-b2c9-4987-a46d-79d4cfedc292 url admin@192.168.110.10:/
Session: 325fede9-b2c9-4987-a46d-79d4cfedc292
Request:
        Capture host: host-29
        Vnic: 501c7db8-e74f-3aac-86ef-99347b1b95a7.000
        Capture point: vnic
        Capture direction: output
Session file: /tmp/pktcap/325fede9-b2c9-4987-a46d-79d4cfedc292.pcap
Session status: finished
Begin SCP:
Could not create directory '/var/empty/.ssh'.
The authenticity of host '192.168.110.10 (192.168.110.10)' can't be established.
RSA key fingerprint is SHA256:I9aPF/MDq6ImYqe1ScH2uiyNVcJNNcT4Uu+Hg99SxzI.
Are you sure you want to continue connecting (yes/no)?
Failed to add the host to the list of known hosts (/var/empty/.ssh/known_hosts).
admin@192.168.110.10's password:
nsxmgr-01a#

 

Now the admin has both .pcap files on hers/his desktop and can open and merge them together with Wireshark (details covered in the demo linked at the beginning of this post).

There is one final step pending, which is deleting the capture information from NSX Manager since we don’t need it anymore. We use the debug packet capture clear command and we issued it twice, once for each session:

nsxmgr-01a# debug packet capture clear session fb39d938-0602-486f-8f3f-c4987f9a06ef
Session: fb39d938-0602-486f-8f3f-c4987f9a06ef
Request:
        Capture host: host-29
        Vnic: 501c7db8-e74f-3aac-86ef-99347b1b95a7.000
        Capture point: vnic
        Capture direction: input
Session file: /tmp/pktcap/fb39d938-0602-486f-8f3f-c4987f9a06ef.pcap
Session status: deleted
nsxmgr-01a#
nsxmgr-01a#
nsxmgr-01a# debug packet capture clear session 325fede9-b2c9-4987-a46d-79d4cfedc292
Session: 325fede9-b2c9-4987-a46d-79d4cfedc292
Request:
        Capture host: host-29
        Vnic: 501c7db8-e74f-3aac-86ef-99347b1b95a7.000
        Capture point: vnic
        Capture direction: output
Session file: /tmp/pktcap/325fede9-b2c9-4987-a46d-79d4cfedc292.pcap
Session status: deleted
nsxmgr-01a#

 

Finally, we confirm they have been deleted with the command show packet capture sessions:

nsxmgr-01a# show packet capture sessions
Sessions:
======================================
Started session: 0
Stopped session: 0
Finished session: 0
Error session: 0
nsxmgr-01a#

 

Anyway, don’t worry if you forget the cleaning steps as NSX will automatically delete stopped sessions after a while. 🙂

 

I hope you have enjoyed the post, if you want to see a demo of how to take a packet trace of a VM using this procedure, just have a look at this video: https://goo.gl/w9hx1r

 

Thanks for reading!

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s