Azure WS2022 Virtual Machine: Unable to complete SMB transfers due to dup ack storm / connection resets

Hello Justin Bakelaar (Protiviti Inc)

Thanks for sharing the logs and the Wireshark capture that makes this much easier to diagnose.

Looking at your packet capture, the pattern is pretty clear: the SMB server (port 445) is advertising a very small receive window (Win=255), and you're seeing a cascade of TCP Duplicate ACKs (#1 through #10+) as the client keeps retransmitting into that near-zero window. Eventually the server sends a TCP RST to tear down the connection entirely. This is a classic receive-buffer exhaustion / TCP window scaling problem, not a routing or firewall issue.

Step 1 – Check TCP Receive Window Autotuning

The Win=255 in your capture strongly suggests TCP autotuning has been disabled or is stuck at a restricted level on the SMB server (or something in the path is clamping it). Run this on the server VM:

netsh int tcp show global

If 'Receive Window Auto-Tuning Level' shows as 'disabled' or 'highlyrestricted', that's your culprit. Set it back to normal:

netsh int tcp set global autotuninglevel=normal

This is the most common cause of the exact Win=255 symptom you're seeing in Azure VMs. Reference: TCP/IP Performance Known Issues – Windows Server

https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/tcpip-performance-known-issues

Step 2 – Verify RSS and NIC Offload Settings

In your Wireshark trace, the 'TCP Previous segment not captured' alongside the Dup ACKs suggests some segments may be getting dropped or reordered at the NIC level — which can happen when Receive Side Scaling (RSS) or Receive Segment Coalescing (RSC) isn't configured correctly on an Azure VM.

Check and enable these on both VMs:

# Check current state

Get-NetAdapterRss

Get-NetAdapterAdvancedProperty | Where-Object DisplayName -like '*Coalescing*'

# Enable RSS if needed

Enable-NetAdapterRss -Name '<adapter-name>'

Note: the hardware RSC offload is intentionally disabled once a NIC is attached to a vSwitch in Azure VMs that's expected. The vSwitch-level RSC should be active instead. Reference: RSC in the vSwitch

https://learn.microsoft.com/en-us/windows-server/networking/technologies/hpn/rsc-in-the-vswitch

For general NIC offload guidance for SMB: Slow SMB File Transfer Speed

https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/slow-smb-file-transfer

Step 3 – Confirm SMB Client MTU and Throttling

While you're at it, make sure large MTU is enabled and bandwidth throttling is off on the SMB client side:

Set-SmbClientConfiguration -EnableBandwidthThrottling 0 -EnableLargeMtu 1

Step 4 – Review the Multiple Administrators / SMB Session Conflict

Your original post mentions 'multiple administrators connecting to the same server.' SMB has session limits and can exhibit unexpected resets when multiple admin sessions compete for the same named pipe or share connection. Check active SMB sessions:

Get-SmbSession

Get-SmbOpenFile

If you're seeing stale sessions from previous disconnects, clear them and test again. Also review SMB server event logs (Event Viewer > Applications and Services Logs > Microsoft > Windows > SMBServer > Operational) for any session teardown events correlated with your RST timestamps.

Step 5 – TCP/IP Performance Tuning for Azure VMs

As a broader reference, Microsoft's TCP/IP performance tuning guide for Azure VMs is worth a read — it covers window scaling, SACK, RTT effects, and how Dup ACKs / retransmits should be interpreted in Azure environments:

https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-tcpip-performance-tuning

Actions to check:

1. Run netsh int tcp show global — fix autotuning if disabled
2. Verify RSS is enabled on both client and server NICs
3. Set SMB client MTU and throttling settings
4. Clear stale SMB sessions if multiple admins are involved
5. Re-run the Wireshark capture after each change to confirm Win size is growing normally

If after all of the above the Win=255 issue persists, it would be worth capturing a simultaneous trace on both sides (client and server) to confirm whether the small window is being set by the server OS or being manipulated in-path.

Thanks,
Manish.

Thanks Manish.

• Step 1 was already set correctly (Receive Window Auto-Tuning Level : normal).
• Step 2 and 3 match current settings which were updated during troubleshooting.
• Step 4 shouldn't apply since a new share was created just for this troubleshooting (1 client connection) but was also reproducible against an on-prem Isilon cluster.

I'll take a look at the performance tuning doc and perform another capture to focus in on the Win=255 issue.

Thanks Justin

We will be waiting for your inputs

Hello Manish, we are seeing similar captures even after redeploying and disabling accelerated networking. Please advise on next steps.

Hello James

Just wanted you to try this solution once

Thanks for the update and for testing those steps. The fact that the issue persists even after a full redeployment and with Accelerated Networking disabled is actually a very important data point. It tells us two things:

1. This is NOT a NIC driver or SR-IOV issue (AN being disabled rules that out completely)
2. The Win=255 / TCP RST pattern is originating from either the OS TCP stack on the server VM itself, or something in the network path (NSG, Azure Load Balancer, or a middlebox) that survives a redeployment

Step 1 – Confirm the TCP Autotuning State on the Freshly Deployed VM

This is the most important check right now. Even on a freshly deployed WS2022 VM, autotuning can be set to 'disabled' or 'highlyrestricted' if it was baked into the image, a GPO is pushing it, or a third-party agent (antivirus, monitoring, security) is overriding it.

Run this on the SMB server VM as Administrator:

netsh interface tcp show global

Share the output here. We specifically need to see the 'Receive Window Auto-Tuning Level' line. If it says anything other than 'normal' or 'experimental', that is your root cause regardless of how many times the VM is redeployed.

To fix it:

netsh interface tcp set global autotuninglevel=normal

For WS2022, you can also use PowerShell (preferred since registry-based settings are deprecated in Server 2019+):

Get-NetTCPSetting | Select SettingName, AutoTuningLevelLocal

Set-NetTCPSetting -SettingName Internet -AutoTuningLevelLocal Normal

Ref: Network Adapter Performance Tuning in Windows Server

https://learn.microsoft.com/en-us/windows-server/networking/technologies/network-subsystem/net-sub-performance-tuning-nics

Step 2 – Check VMQ (Virtual Machine Queue)

On Azure VMs, Virtual Machine Queue (VMQ) should be used instead of standard RSS, and it must be correctly enabled. A misconfigured VMQ is a known cause of TCP receive window exhaustion on Windows Server.

# Check VMQ state

Get-NetAdapterVmq

# Enable if needed

Enable-NetAdapterVmq -Name "<adapter-name>"

Ref: TCP/IP Performance Known Issues – Windows Server

https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/tcpip-performance-known-issues

Step 3 – Check for Windows Filtering Platform (WFP) Interference

Antivirus, endpoint security agents, or host-based firewalls that use the Windows Filtering Platform can silently clamp TCP receive windows — and they survive redeployment if they're part of your VM image or auto-installed via extensions. This is a very common culprit in persistent Win=255 cases.

To check active WFP filters:

netsh wfp show filters

Or temporarily disable Windows Firewall and any third-party security agents on both VMs, reproduce the transfer, and check if the Wireshark capture still shows Win=255. If the window grows normally with security agents off, that's your culprit.

Step 4 – Run a Simultaneous Dual-Sided Packet Capture

This is the most diagnostic thing you can do right now. A capture taken only on the client side can't tell us whether Win=255 is being set by the server OS or being modified in-path. We need to see both ends at the same time.

Use Azure Network Watcher to capture simultaneously on both the client and server VMs:

• Azure Portal → Network Watcher → Packet Capture
• Start a capture on both VMs filtered to port 445
• Reproduce the transfer failure
• Download both .cap files and open them side by side in Wireshark

What to look for:

• If the server-side capture shows Win=255 being advertised by the server itself → OS-level issue (autotuning / WFP)
• If the server-side capture shows a normal window but the client-side capture shows Win=255 → something in the network path is modifying the TCP header

Ref: Network Watcher Packet Capture – Inspect and Analyze Captures

https://learn.microsoft.com/en-us/azure/network-watcher/network-watcher-deep-packet-inspection

Ref: Azure Network Watcher Overview

https://learn.microsoft.com/en-us/azure/network-watcher/network-watcher-overview

Step 5 – Check NSG Rules and VNet Flow Logs

Since the issue persists across redeployments, it's worth verifying that no NSG rule at the subnet level is interfering with established TCP flows (some NSG configurations can cause unexpected RSTs on long-lived sessions).

Use IP Flow Verify to confirm port 445 traffic is clean end-to-end:

   az network watcher test-ip-flow \
      --direction Inbound \
      --protocol TCP \
      --local <server-IP>:445 \
      --remote <client-IP>:* \
      --vm <server-vm-name> \
      --resource-group <rg-name>

Also enable VNet Flow Logs if you haven't already — they can surface blocked or reset flows that don't show up in standard monitoring:

https://learn.microsoft.com/en-us/azure/network-watcher/network-watcher-overview

What We Need from You Next

1. Share the output of: netsh interface tcp show global (from the server VM)
2. Share the output of: Get-NetTCPSetting
3. Confirm whether any antivirus/endpoint security agents are installed on the VM
4. Run a dual-sided Network Watcher packet capture and share both files

We're getting close to isolating this the dual-sided capture in particular should give us a definitive answer on where the Win=255 is originating. Hang tight and let us know what you find!

Thanks,
Manish.