This post is a part of the nine-part “What’s New in Windows Server & System Center 2012 R2” series that is featured on Brad Anderson’s In the Cloud blog. Today’s blog post covers new diagnostic features to quickly isolate network problems, and how they apply to the larger topic of “Transform the Datacenter.” To read that post and see the other technologies discussed, read today’s post: “Delivering Infrastructure as a Service (IaaS).”
As described in Transforming your datacenter – networking, Cloud Scale Performance and Diagnosability is a key investment area for this release. Service providers cannot afford downtime due to misconfigurations and need to be able to isolate a problem as quickly as possible.
We find that developers, administrators, and service providers often have difficulty identifying the source of networking problems when they occur. This becomes even more problematic as we consider the addition of new technologies such as Hyper-V Network Virtualization (HNV) that service providers will need to diagnose.
In Windows Server 2012 R2, we make some of the routine diagnostics tasks easier with PowerShell. In this blog, we are going to walk you through basic network management with PowerShell and then dive deeper into a few new features that improve diagnostics and troubleshooting of network connections.
Viewing and understanding a system’s configuration is the first step in being able to configure the basic network settings through PowerShell. In this introduction, that’s exactly what we hope to teach you by introducing you to a few PowerShell cmdlets that describe in-depth networking information. We will also show you a shortcut with a PowerShell cmdlet, Get-NetIPConfiguration and its alias GIP. GIP is a one stop shop when it comes to your computer’s networking configuration. GIP combines elements of multiple PowerShell cmdlets that give detailed information and acts as an “orchestrator” to give you a comprehensive view of how your system is setup. To fully understand GIP, we will first go over the underlying cmdlets that it uses.
The first cmdlet is Get-NetAdapter. This cmdlet retrieves the basic network adapter properties for all of the physical adapters connected to the system. It details the type of adapter, adapter name, the MAC address, link speed and its current status. By default only visible adapters are returned. This information can be useful when trying to trouble shoot a problem on a remote computer. This will tell you the adapters on the system so you know what you’re working with.
PS C:\WINDOWS\system32> Get-NetAdapter Name InterfaceDescription ifIndex Status MacAddress LinkSpeed ---- -------------------- ------- ------ ---------- --------- Ethernet 5 Broadcom BCM5709C Ne...#36 7 Disconnected 00-10-18-44-06-7A 0 bps Ethernet 2 Broadcom BCM5709C Ne...#35 4 Disconnected 00-10-18-44-06-78 0 bps Ethernet 4 Broadcom NetXtreme Gigabit 6 Up B8-AC-6F-82-68-DB 1 Gbps Ethernet 3 Intel(R) Ethernet Ser...#2 5 Disconnected 00-1B-21-8E-EB-F5 0 bps Ethernet Intel(R) Ethernet Serve... 3 Disconnected 00-1B-21-8E-EB-F4 0 bps
If you are running in a Hyper-V environment, there is also a cmdlet to gather information for the VM NICs on that host. A VM NIC is the equivalent of a network adapter for a VM.
PS C:\Users\Administrator> Get-VMNetworkAdapter -VMName DemoVM1 Name IsManagementOs VMName SwitchName MacAddress Status IPAddresses ---- -------------- ------ ---------- ---------- ------ ----------- Network Adapter False DemoVM1 BackToBack 00155D3AE401 {Ok} {192.168.0.3}
In the example above we choose a specific VM but you can also search for all VM NICs by specifying the wildcard or ‘*’ character after the cmdlet. An example of this cmdlet is given below.
PS C:\Users\Administrator> Get-VMNetworkAdapter * Name IsManagementOs VMName SwitchName MacAddress Status IPAddresses ---- -------------- ------ ---------- ---------- ------ ----------- Network Adapter False DemoVM2 BackToBack 00155D3AE403 {Ok} {192.168.0.4} Network Adapter False DemoVM1 BackToBack 00155D3AE401 {Ok} {192.168.0.3}
Get-NetIPAddress will return the IP addresses that are configured on that system system for both IPv6 and IPv4. This can be run inside a VM or on the host.
PS C:\> Get-NetIPAddress IPAddress : fe80::bf7f%7 InterfaceIndex : 7 InterfaceAlias : Ethernet 5 AddressFamily : IPv6 Type : Unicast PrefixLength : 64 PrefixOrigin : WellKnown SuffixOrigin : Link AddressState : Deprecated ValidLifetime : Infinite ([TimeSpan]::MaxValue) PreferredLifetime : Infinite ([TimeSpan]::MaxValue) SkipAsSource : False PolicyStore : ActiveStore IPAddress : 192.168.191.127 InterfaceIndex : 7 InterfaceAlias : Ethernet 5 AddressFamily : IPv4 Type : Unicast PrefixLength : 16 PrefixOrigin : WellKnown SuffixOrigin : Link AddressState : Tentative ValidLifetime : Infinite ([TimeSpan]::MaxValue) PreferredLifetime : Infinite ([TimeSpan]::MaxValue) SkipAsSource : False PolicyStore : ActiveStore
These cmdlets provide a wealth of detailed information.
GIP is an alias for Get-NetIPConfiguration and is meant to provide concise output that will give an overview of the system’s network configuration without having to dig through the details of the cmdlets explained above. GIP provides the IPv4/IPv6 addresses, gateways, interface alias/description/index and DNS servers of adapters with an up status and only interface information of adapters that are down.
PS C:\> GIP InterfaceAlias : Ethernet 4 InterfaceIndex : 6 InterfaceDescription : Broadcom NetXtreme Gigabit Ethernet NetProfile.Name : ntdev.corp.microsoft.com IPv6Address : 2001:4898:1a:3:f0d3:dc16:89c6:86eb IPv4Address : 192.31.25.52 IPv6DefaultGateway : fe80::8a75:56ff:fe3d:7380 IPv4DefaultGateway : 192.31.25.1 DNSServer : 158.55.16.178
You can use the –Detailed flag with GIP to show more information.
PS C:\> GIP -Detailed ComputerName : GS-BDESKTOP InterfaceAlias : Ethernet 4 InterfaceIndex : 6 InterfaceDescription : Broadcom NetXtreme Gigabit Ethernet NetAdapter.LinkLayerAddress : B8-AC-6F-82-68-DB NetAdapter.Status : Up NetProfile.Name : work.microsoft.com NetProfile.NetworkCategory : DomainAuthenticated NetProfile.IPv6Connectivity : Internet NetProfile.IPv4Connectivity : Internet IPv6Address : 2001::86eb IPv6TemporaryAddress : 2001::e7b4 IPv6LinkLocalAddress : fe80::86eb%6 IPv4Address : 192.31.25.52 IPv6DefaultGateway : fe80::8a75:56ff:fe3d:7380 IPv4DefaultGateway : 192.31.25.1 NetIPv6Interface.NlMTU : 1500 NetIPv4Interface.NlMTU : 1500 NetIPv6Interface.DHCP : Enabled NetIPv4Interface.DHCP : Enabled DNSServer : 158.55.16.178
Like most PowerShell cmdlets, Get-NetIPConfiguration returns rich scriptable objects. For example, the default route is accessible directly.
PS C:\WINDOWS\system32> $Object = GIP 'Ethernet 4' PS C:\WINDOWS\system32> $Object.IPv4DefaultGateway ifIndex DestinationPrefix NextHop RouteMetric PolicyStore ------- ----------------- ------- ----------- ----------- 6 0.0.0.0/0 172.31.232.1 0 ActiveStore
Windows Server 2012 R2 supports a new cmdlet for testing ICMP and TCP connectivity, called Test-NetConnection (TNC).
Ping, nslookup and tracert are basic tools of the networking trade. For many scenarios however, TNC is a more scriptable and usable alternative to these tools. It provides much more information in a single cmdlet to help troubleshoot a connection quickly.
Here is an example of using TNC to test connectivity to www.xbox.com.
PS C:\WINDOWS\system32> tnc www.Xbox.com ComputerName : www.xbox.com RemoteAddress : 2600:1409::1ac InterfaceAlias : Ethernet 4 SourceAddress : 2001::e7b4 PingSucceeded : True PingReplyDetails (RTT) : 2 ms
The output object of TNC includes the relevant WMIv2 objects of the local machine. This means you can script against them directly. Below is an example of using TNC to rename the network adapter that is used by default to reach www.xbox.com.
PS C:\WINDOWS\system32> tnc www.xbox.com | Rename-NetAdapter -NewName "ConnectsToXbox" PS C:\WINDOWS\system32> $Object.NetAdapter Name InterfaceDescription ifIndex Status MacAddress LinkSpeed ---- -------------------- ------- ------ ---------- --------- ConnectsToXbox Broadcom NetXtreme G... 6 Up B8-AC-6F-82-68-DB 1 Gbps
Using the –TraceRoute flag will return the list of hosts on the path to the specified target.
PS C:\WINDOWS\system32> tnc www.xbox.com -TraceRoute ComputerName : www.xbox.com RemoteAddress : 2600:1409:a:18f::1ac InterfaceAlias : ConnectsToXbox SourceAddress : 2001::e7b4 PingSucceeded : True PingReplyDetails (RTT) : 2 ms TraceRoute : 20013::1 2001::2 2001::6c:1 TimedOut 2001:4898:8000:2:ff::36 TimedOut TimedOut TimedOut TimedOut 2600:1409:a:18f::1ac
While testing ICMP connectivity is a common troubleshooting step, most of the time you’re interested in resolving a TCP connectivity issue. TNC enables you to test connectivity to a specified remote host and port.
PS C:\WINDOWS\system32> tnc www.bing.com -Port 80 ComputerName : www.bing.com RemoteAddress : 204.79.197.200 RemotePort : 80 InterfaceAlias : ConnectsToXbox SourceAddress : 192.31.25.5 PingSucceeded : True PingReplyDetails (RTT) : 0 ms TcpTestSucceeded : True
TNC has some common keywords to simplify selection of which TCP port to test. For example, to test remote desktop connectivity, use the RDP keyword. In the example below you can see that the ping succeeded but the TCP test to remote port 3389 failed.
PS C:\WINDOWS\system32> tnc HomeBase RDP WARNING: TCP connect to HomeBase:3389 failed ComputerName : HomeBase RemoteAddress : 192.168.0.5 RemotePort : InterfaceAlias : ConnectsToXbox SourceAddress : 192.31.25.5 PingSucceeded : True PingReplyDetails (RTT) : 0 ms TcpTestSucceeded : False
In the prior example, I couldn’t remote desktop to my home server, HomeBase. Let’s use the Detailed flag to see more information.
PS C:\> tnc HomeBase RDP -InformationLevel Detailed WARNING: TCP connect to HomeBase:3389 failed ComputerName : HomeBase RemoteAddress : 10.195.58.228 RemotePort : 3389 AllNameResolutionResults : 10.195.58.228 MatchingIPsecRules : Msit-Ipsec-Win8/WS8-Domain-CorpnetIpv4-Authip NetworkIsolationContext : Private Network InterfaceAlias : ConnectsToXbox SourceAddress : 192.31.25.5 NetRoute (NextHop) : 192.31.25.1 PingSucceeded : True PingReplyDetails (RTT) : 0 ms TcpTestSucceeded : False
The MatchingIPSecRules field tells us which IPsec rules are attempting to secure the connection, in this case the IPsec rules that Microsoft’s IT department has deployed for DirectAccess. Because this is PowerShell, the property in fact is a rich WMI object.
The NetworkIsolationContext field tells us which capability Windows store apps need to use to access the resource. In order for Windows store apps to access HomeBase, they need to declare the Private Network capability.
If you’re wondering why DirectAccess isn’t working, it’s because I haven’t provided my smartcard credentials to the system.
PS C:\WINDOWS\system32> Get-DAConnectionStatus Status : ActionableError Substatus : StrongAuthorizationCredentialsNeeded
There are many elements of TNC that aid with scripting. Some key features are specified below:
1. If DNS resolution fails to the specified host, an exception is thrown and can be caught.
2. Using the InformationLevel parameter, you can have TNC run in a “quiet” mode. This means that TNC simply returns a Boolean indicating if the connection succeeded or not.
if(TNC www.bing.com –InformationLevel Quiet) {}
3. When using TNC to test TCP connectivity, if the connection succeeds, you can access the TCP socket using the TCPClientSocket property. This provides the connected .NET socket for further scripting.
4. The route, network adapter, and source IP address, used for connectivity, are all accessible as WMI objects embedded in TNC’s output.
5. GIP and TNC produce objects that can be serialized into XML files. You can send the “rich” objects over emails, instead of simply cutting and pasting the output.
PS C:\WINDOWS\system32> tnc www.xbox.com HTTP | export-clixml C:\Output.xml PS C:\WINDOWS\system32> Import-Clixml C:\Output.xml ComputerName : www.xbox.com RemoteAddress : 2600:1409:a:181::1ac RemotePort : 80 InterfaceAlias : ConnectsToXbox SourceAddress : 2001::e7b4 PingSucceeded : True PingReplyDetails (RTT) : 2 ms TcpTestSucceeded : True
Windows Server 2012 R2 also helps with diagnosing connectivity issues when access to VMs is not available like in service provider environments. In service provider environments, it will be common to see tenant overlays on the physical network using Hyper-V Network Virtualization (HNV). With HNV, customers can easily move their subnets to the cloud while preserving their existing IP addresses (called a Customer Address or CA), and topology into the cloud so that existing services continue to work unaware of the underlying fabric’s address space (called a, Provider Address or PA), used by the service provider. As a result of this isolation and indirection, testing connectivity between VMs is challenging and tools like TNC may not always suffice.
Diagnosing connectivity for the service provider and tenant networks, however, remains important in order to validate infrastructure setup or deployment configuration. To help diagnose these issues Windows Server 2012 R2 introduces two new tools: PA Ping and Test-VMNetworkAdapter.
PA Pings allows admins to validate connectivity in the physical infrastructure that the tenant networks are overlaid on. The system injects ICMP packets above physical NIC of the source VM and waits for an echo response from the physical NIC of the destination VM. This enables the ability of sending an ICMP packet from one PA to another. This tool is available in the traditional ping tool with the –p parameter:
C:\> ping –p 1.2.4.15 1.2.3.20 Pinging 1.2.3.20 with 32 bytes of data Reply from 1.2.3.20: bytes=32 time=1ms TTL=128 Reply from 1.2.3.20: bytes=32 time<1ms TTL=128 Reply from 1.2.3.20: bytes=32 time<1ms TTL=128 Reply from 1.2.3.20: bytes=32 time<1ms TTL=128 Ping statistics for 1.2.3.30: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 1ms, Average = 0ms
Test-VMNetworkAdapter allows admins to diagnose connectivity issues in the tenant network. The challenge here is that admins may not have direct access to the VMs and therefore cannot run tools like TNC from within the VM. Test-VMNetworkAdapter injects ICMP packets on the port that the VM is attached to and awaits an echo response from the receiver VM:
PS C:\> Test-VMNetworkAdapter –VMName Helen –sender –SenderIPAddress 10.20.20.5 –ReceiverIPAddress 10.20.20.6 –NextHopMacAddress f4-ce-46-2d-98-16 –VMNetworkAdapterName HelenNic RoundTripTime : 16 milliseconds
If this command succeeds, this can indicate that the problem is in the VM, and not in the tenant network.
If the NextHopMacAddress is not known, use Select-NetVirtualizationNextHop cmdlet to look up several parameters like NextHopMACAddress from the HNV database:
PS C:\> Select-NetVirtualizationNextHop –DestinationCustomerAddress 10.20.20.6 –SourceCustomerAddress 10.20.20.5 –SourceVirtualSubnetID 5001 SourceCustomerAddress : 10.20.20.5 DestinationCustomerAddress : 10.20.20.6 SourceVirtualSubnetID : 5001 NextHopAddress : 0.0.0.0 SourceMACAddress : fece462d9814 NextHopMACAddress : fece462d9816
For business reasons you may want to avoid sending ICMP ping packets to their tenant VMs. To do so, use the receiver parameter to enable an ICMP responder running on the switch port of the VM:
PS C:\> Test-VMNetworkAdapter –VMName Helen –receiver –SenderIPAddress 10.20.20.5 –ReceiverIPAddress 10.20.20.6 –VMNetworkAdapterName HelenNic
The control channel inspects all packets and echoes back ICMP packets that contain the sequence number 100 in their payload. This number can be configured using the –SequenceNumber parameter. Be sure to use the same sequence number on both VMs so that the receiver may intercept and echo back the correct ICMP packets.
Let’s take a look at an example scenario that merges the use of these cmdlets: Brian has just deployed HNV on his network. His topology consists of multiple hosts with multiple VMs on each host. He wants to validate connectivity between each VM and ensure HNV is deployed correctly.
First, he will use PA Ping (ping –p) to validate connectivity over the physical infrastructure between each VM. Then he will use Test-VMNetworkAdapter with –sender and –receiver parameters to validate connectivity between each VM. In doing so he will validate that the indirection introduced by HNV has resulted in the mapping as configured at the time of deployment. Finally, he can use the same script with Test-VMNetworkAdapter without the –receiver parameter to validate correct firewall configuration in the operating system on the VMs.
By doing this, Brian can validate the correct configuration of HNV on his network and ensure connectivity between all VMs. If a customer were to report a connectivity problem, Brian can quickly resolve it back to the customer saying that he expects the problem is in the customer VM! The customer can now use TNC to detect further problems.
Fast diagnosability is key towards driving down operational costs in the cloud. With the added diagnostics capabilities in Windows Server 2012 R2, we hope you are able to isolate issues more quickly.
Give it a try, and let us know what you think!
Gabriel Silva, Program Manager, Windows Core Networking Team
Christopher Palmer, Program Manager, Windows Core Networking Team
Suhail Khalid, Program Manager, Windows Core Networking Team
To see all of the posts in this series, check out the What’s New in Windows Server & System Center 2012 R2 archive.
This is awesome. A new way to do old things, but simpler. I just need to memorise these commands and get rid of ipconfig, ping, and tracert!
I would suggest some context in these commands. The Ip addresses and VM names are random for example on ping -p I THINK this runs in the VM, but is the first IP address the customer address and the provider address. It also looks like you need to apply some fixes to get the ping -p test-vmnetworkadapter seems to not document the needed sequence number parm