Source IP Address Preference with Multiple IPs on a NIC

Something I'm finding myself doing more and more lately is using multiple IP addresses on a single NIC for a Windows server. The reasons vary, but it's generally in order to support a single server running 2 different services on the same port. This can happen for Lync with your Edge servers (or for skirting the reverse proxy requirement on Front-Ends), or with Exchange when creating multiple receive connectors on a server.

A behavior that changed with the introduction of Server 2008 is that the source IP address on a NIC will always be the lowest numerical IP. So that whole idea of your primary IP being the first one you put on the NIC - throw that idea out the window.

For example, let's say we build a new Exchange server and configure the NIC with IP 10.0.0.100. This IP is registered in DNS and the server uses this IP as the source when communicating with other servers. Our fantastic network administrator has also created a NAT rule on the firewall to map this IP to a particular public IP for outbound SMTP so that our PTR lookups match up.

But now we want to add another IP for a custom receive connector and the network admin hands you a free IP which happens to be 10.0.0.50. You add this as an additional IP on the NIC and voila - you have a couple issues:

  • You just registered two names for the same server in DNS if dynamic registration is enabled.
  • Your server is now sending all outbound traffic from 10.0.0.50! (because 50 is lower than 100)

One of these is easily solved - just turn off dynamic registration and manually create the DNS records for the server. The other one is a little trickier because Server 2008 and 2008 R2 will still be sending traffic as the 10.0.0.50 IP. In the case of Exchange, this could create some ugliness for outgoing SMTP because now your firewall is not NATing to the correct public IP and you start bouncing mail due to PTR lookup failures.

Fortunately, we have a way to tell Windows not to use the lower numbered IP as a source address by adding the IP via the netsh.exe command. For Server 2008 SP2 and 2008 R2 RTM we need to apply a hotfix first. 2008 R2 SP1 included this fix by default so it is no longer required. Without the hotfix or SP1 you'll find netsh.exe does not display or recognize the special flag.

Hotfix Downloads:

The key to this is the IP address must be added via netsh.exe with a particular flag. So if you've already added the IP address via the GUI you'll need to remove it first. After that, use this command to add the secondary IP:

netsh int ipv4 add address "Local Area Connection" 1.2.3.4/24 SkipAsSource=true

The SkipAsSource flag does two things - first, it instructs Windows not to use this IP as a source IP for outgoing traffic. And secondly, it prevents the registration of this IP in DNS if dynamic registration is enabled. Two birds with one stone!

You can always view the status of the IPs and their SkipAsSource status with the following command:

netsh int ipv4 show ipaddresses level=verbose

Communicator and Damaged Address Book Files

The past few days I spent wrestling an address book server issue in OCS and I wanted to share the solution. The quick version: make sure you have your server side and client side hotfix revisions match up.

If you want the whole story…the specific details of this case involved a Front-End server which had the OCS 2007 R2 April hotfixes, but the MOC clients had the July hotfixes applied. The issue first manifested itself with clients reporting ABS damaged:

Communicator cannot synchronize with the corporate address book because the corporate address book file appears to be damaged. Contact your system administrator with this information.

We resolved this by deleting the entire contents of the address book file share and forcing a resync of the address book. We also deleted GalContacts.db from a few user workstations, but later found the client error actually disappeared on its own without removing the file.

Things hummed along nicely for a week or so until a large amount of users (600+) were enabled for OCS one Friday evening. The following Monday previously enabled pilot users were reporting they still didn’t have a SIP URI in their address books for the mass-enabled users. The GalContacts.db file was also still showing a timestamp from the day the mass change occurred, indicating they had not downloaded an update yet.

We took a peek at the Front-End logs and it appeared to be generating address book files correctly. The odd thing was in looking at the IIS logs we actually saw quite a few 404 errors of MOC clients trying to request delta files that did not exist. Other users showed successful downloads of the latest delta files which should have included the changes, but they weren’t being applied to their local GalContacts.db for some reason. I also saw those same clients registering a success end up using the fallback logic and downloading older address book files even though they had the newer versions. Very, very strange. Any client we deleted GalContacts.db on would pull down the latest full address book with no issues. The clients looking for deltas that didn't exist we probably caused by deleting the address book files previously.

Side tip when looking at the IIS logs: Full files start with F-xxxx and delta files follow a D-xxxx-xxxx naming convention. Also, .lsabs files are used by MOC while .dabs files are used by Tanjay devices.

At that point I noticed the mismatch in server (April hotfixes) vs. client (July hotfixes) versions and suggested we get the latest fixes installed on all sides. While that suggestion made its way through change control procedures we opened a PSS case with Microsoft to hit the problem from another angle. The engineer we spoke with immediately blurted out that we needed to match the hotfix versions as soon as we described the behavior. It sounded to me like this was one he had heard before or was familiar with so while we didn’t have a second approach this definitely helped accelerate the change control ticket to an authorized state. After we fully patched the Front-End using the new ServerUpdateInstaller (a lifesaver), applied the back-end database hotfix, and installed the client October hotfix the address book went back to functioning properly. There were a couple of users that needed to delete the GalContacts.db before everything went back to normal, but most of them picked it up without intervention.

As for root cause, the KB 972403 article actually does reference applying both the MOC and server fix together, but the July server hotfix document doesn’t describe this behavior or even mention it. Personally, I think the underlying issue was having the 3.5.6907.37 hotfix on clients while the abserver.exe file was still at 3.5.6907.0. In any case, I learned a lot more about the ABS than I ever cared to, but it was great information that will surely help in the future.