Source IP Address Preference with Multiple IPs on a NIC

Something I’m finding myself doing more and more lately is using multiple IP addresses on a single NIC for a Windows server. The reasons vary, but it’s generally in order to support a single server running 2 different services on the same port. This can happen for Lync with your Edge servers (or for skirting the reverse proxy requirement on Front-Ends), or with Exchange when creating multiple receive connectors on a server.

A behavior that changed with the introduction of Server 2008 is that the source IP address on a NIC will always be the lowest numerical IP. So that whole idea of your primary IP being the first one you put on the NIC – throw that idea out the window.

For example, let’s say we build a new Exchange server and configure the NIC with IP 10.0.0.100. This IP is registered in DNS and the server uses this IP as the source when communicating with other servers. Our fantastic network administrator has also created a NAT rule on the firewall to map this IP to a particular public IP for outbound SMTP so that our PTR lookups match up.

But now we want to add another IP for a custom receive connector and the network admin hands you a free IP which happens to be 10.0.0.50. You add this as an additional IP on the NIC and voila – you have a couple issues:

  • You just registered two names for the same server in DNS if dynamic registration is enabled.
  • Your server is now sending all outbound traffic from 10.0.0.50! (because 50 is lower than 100)

One of these is easily solved – just turn off dynamic registration and manually create the DNS records for the server. The other one is a little trickier because Server 2008 and 2008 R2 will still be sending traffic as the 10.0.0.50 IP. In the case of Exchange, this could create some ugliness for outgoing SMTP because now your firewall is not NATing to the correct public IP and you start bouncing mail due to PTR lookup failures.

Fortunately, we have a way to tell Windows not to use the lower numbered IP as a source address by adding the IP via the netsh.exe command. For Server 2008 SP2 and 2008 R2 RTM we need to apply a hotfix first. 2008 R2 SP1 included this fix by default so it is no longer required. Without the hotfix or SP1 you’ll find netsh.exe does not display or recognize the special flag.

Hotfix Downloads:

The key to this is the IP address must be added via netsh.exe with a particular flag. So if you’ve already added the IP address via the GUI you’ll need to remove it first. After that, use this command to add the secondary IP:

netsh int ipv4 add address "Local Area Connection" 1.2.3.4/24 SkipAsSource=true

The SkipAsSource flag does two things – first, it instructs Windows not to use this IP as a source IP for outgoing traffic. And secondly, it prevents the registration of this IP in DNS if dynamic registration is enabled. Two birds with one stone!

You can always view the status of the IPs and their SkipAsSource status with the following command:

netsh int ipv4 show ipaddresses level=verbose

OCS Create Pool Wizard Error: Invalid database parameter

Recently I had a project where we were moving the OCS databases to a new clustered SQL 2008 (R1) with SP2 Back-End and ran into a lovely new error I’d never seen before – also not seen before anywhere on Google!

For starters, we followed the steps outlined on Technet. After we had successfully detached and attached all databases and ran the LCSCMD.exe step, we launched the Create Pool wizard and attempted to plug in the info for the new SQL cluster. We got this error back:

An error occurred during the pool backend detection:

Pool backend discovery failed.

Invalid database parameter.

I double-checked the server name, instance, and FQDN and all looked well. We verified the SQL server was accessible via TCP 1433 and no firewall rules were preventing access, so the error didn’t make a lot of sense. Obviously there was some kind of parameter that the wizard GUI was not cool with. I thought maybe this was the SQL allow updates issue, but that solution had no effect on this error. There was definitely some validation check the UI was failing on against our new DB.

Since I couldn’t locate anyone else with this issue I figured my options were to call PSS and extend this process by a few hours, or pull out the ol’ LCSCMD.exe again and try this operation via command line. The Create Pool wizard really is just collecting a bunch of information and then using it to execute the LCSCMD.exe commands in the background so while doing it manually is not fun, it works just as well.

The entire syntax for LCSCMD.exe can be found on Techet, but here is the command we ended up running. Please note, conferencing archiving was not implemented so that paramter is not present.

LCSCMD.exe /Forest /Action:CreatePool /PoolName:MyOCSPool /PoolBE:MySQLServer.ptown.local\OCSInstance /PoolFQDN:MyOCSPool.ptown.local /InternalWebFQDN:MyOCSPool.ptown.local /ExternalWebFQDN:PublicOCSWebComponents.confusedamused.com /RefDomain:ptown.local /ABOutputlocation:\\\\MyFileServer\AddressBook /MeetingContentPath:\\\\MyFileServer\MeetingContent /MeetingMetaPath:\\\\MyFileServer\MeetingMetadata /AppDataLocation:\\\\MyFileServer\AppData /ClientUpdateLocation:\\\\MyFilerServer\ClientUpdates /DBDataPath:"D:\Databases" /DBLogPath:"L:\Logs" /DynDataPath:"D:\Databases" /DynLogPath:"L:\Logs" /ABSDataPath:"D:\Databases" /ABSLogPath:"L:\Logs" /ACDDataPath:"D:\Databases" /ACDLogPath:"L:\Logs"

After running the command manually it succeeded with absolutely no issues. The new cluster has been running for over a week now without any issues so I think this is an problem specific to the UI. I’m not sure exactly what causes it, but our environment was running SQL 2008 with SP2 on top of a 2008 R2 SP1 operating system.

As a sidenote, this process seems to undo any changes made by the OCS2009-DBUpgrade.msi patches. You’ll need to re-run the patch version which lines up with your FE patch levels before the FE services will be able to start.

Outlook Integration Error in Communicator 2007 R2 when Exchange System Manager is installed

Working on an OCS migration project a few weeks ago I ran into everyone’s favorite error:

There was a problem connecting to Microsoft Office Outlook. Your Outlook profile is not configured correctly. Contact your system administrator with the information.

After double checking the lengthy KB 2373585 article discussing Outlook/Communicator errors and ruling out the usual suspects I was stumped. After some digging around on the workstation I found the user had the Exchange 2003 System Manager and tools installed on the machine. Since the System Manager uses a slightly different version of MAPI components Communicator would generate this error immediately upon signing in.

The solution is to open a command prompt and just run the command: fixmapi.

OCS 2007 R2 Cumulative Update 6 and Stored Procedure Mismatches

Something not mentioned in the release notes of Cumulative Update (CU6) is that there is a dependency on running the new OCS2009-DBUpgrade.msi before any server updates. If you try to run the ServerUpdateInstaller.exe and apply the server updates without first running the database package you may see an error like this:

Event ID: 30968
Source: Live Communications User Services
Details: The component Live Communications User Services reported a critical error: code C3EE78F8 (Enterprise Edition Server successfully registered with the back-end, but a stored procedure version mismatch was detected. The service will not start until this problem is resolved. Cause: The database schema and the Enterprise Edition Server were updated by different installation packages. Resolution: Ensure both the Enterprise Edition Server and back-end were installed or modified by the same installation package. The service has to stop.

Obviously the error verbiage is a bit outdated with references to LCS, but the error is correct – there is a mismatch between the stored procedure versions which makes the Front-End service to fail to start.

To avoid the issue be sure to apply the latest OCS2009-DBUpgrade.msi package before updating any Front-End servers.

The C Stands for Compact

I believe in my last post about damaged Communicator address book files I pointed out that it was a good idea to keep your OCS clients and servers on the same hotfix levels. I would still argue that’s a good thing to do in general, but in my case this wasn’t the actual resolution. While it worked for awhile after a few weeks the damaged address book files error popped up again:

Communicator cannot synchronize with the corporate address book because the corporate address book file appears to be damaged. Contact your system administrator with this information.

All the MOC clients and OCS servers were at the same revision level this time so that wasn’t the problem. Deleting the GalContacts.db and forcing a full download would succeed and the client goes along perfectly happy. Interestingly enough, deleting the GalContacts.db.idx file on a problematic machine would allow the delta file to download successfully so it appears the issue may be with the index file. Anti-virus logs also showed they weren’t trying to clean or repair the file in any way.

I couldn’t find any errors server-side and everything seemed to be functioning properly so I looked at the IIS logs on the Front-End again. Low and behold – the log was huge compared to previous days – about 10x as big. It was filled with many, many requests for downloads of files in the C-xxxx-xxxx.lsabs format which threw me off because the ABS documentation points out that F-xxxx files are fulls, and D-xxxx-xxxx files are delta changes, but has zero mention of the C-xxxx-xxxx files. These IIS requests were also successful downloads, not failures so I wouldn’t have expected clients to have an error, but every user was also repeatedly downloading the same sets of files and then trying to download previous C-xxxx-xxxx files as well.

I took one of the matching C-xxxx-xxxx and D-xxxx-xxxx files (they’ll have the same hex names) and dumped both to text files using abserver.exe –dumpfile to try and compare. Viewing them side-by-side they seemed to have the same content, but in a slightly different order. So it appears they were both delta files, but the C file was about 50% of the D’s size. Odd, but I still had no clue when they would be used over a D because there was zero documentation about the change.

Thanks to a few kind folks on Twitter (@aumblumberg and @MacRS4 ) who went down this same road with Microsoft via a support case previously, I found out the new C files stand for “Compact” and the change was implemented in the July server-side hotfixes. These are also delta files, but compressed in a way supposedly to make them more efficient. In our case (and theirs), it broke the address book downloads completely.

Fortunately, there is a registry key available to prevent clients from trying to use these compact files. This key only applies if you’re using the October Communicator client-side hotfix:

HKLM\Software\Policies\Microsoft\Communicator
Key name: GalUseCompactDeltaFile
Type: DWORD

Possible Values:

  • 0: Do not use compact delta file
  • 1: Use compact delta file (default)
  • 2: Use compact delta file, but do not issue an LDAP query to retrieve the “Title” and “Office” attribute values from Active Directory

You can read more about this registry setting from KB 976985 even though the actual KB is aimed at a different issue with LDAP queries and account lockouts.

I’ll find out today whether this actually fixes the downloads without having to clear out the GalContacts.db file on each client.

It looks like these constant address book changes like this and adding the 0-60 minute download delay are aimed at the larger organizations with a significant address book size, but. I almost feel like these updates are on par with the Resource Kit book providing examples for companies with 100,000+ users in various locations. Great info, but what about the real world? Not everyone using OCS is that big and it would be swell to have guidance around small and medium-sized deployments instead of trying to take those numbers and make them fit. I’d be happy to just let the ABS download the way it used to and leave it alone.

The most frustrating part here has been that this service has been something that traditionally just worked without intervention and instead I’ve been spending hours and hours troubleshooting to figure out what happened because there was nothing mentioned about the change in behavior server or client-side. Maybe there should be some threshold for these ABS disasters optimization changes where they only occur if the full address books are over some value like 10, 20, 50 or 100 MB? Until that happens I’ll be disabling the compact delta files at future OCS deployments to make sure we avoid this problem.

Communicator and Damaged Address Book Files

The past few days I spent wrestling an address book server issue in OCS and I wanted to share the solution. The quick version: make sure you have your server side and client side hotfix revisions match up.

If you want the whole story…the specific details of this case involved a Front-End server which had the OCS 2007 R2 April hotfixes, but the MOC clients had the July hotfixes applied. The issue first manifested itself with clients reporting ABS damaged:

Communicator cannot synchronize with the corporate address book because the corporate address book file appears to be damaged. Contact your system administrator with this information.

We resolved this by deleting the entire contents of the address book file share and forcing a resync of the address book. We also deleted GalContacts.db from a few user workstations, but later found the client error actually disappeared on its own without removing the file.

Things hummed along nicely for a week or so until a large amount of users (600+) were enabled for OCS one Friday evening. The following Monday previously enabled pilot users were reporting they still didn’t have a SIP URI in their address books for the mass-enabled users. The GalContacts.db file was also still showing a timestamp from the day the mass change occurred, indicating they had not downloaded an update yet.

We took a peek at the Front-End logs and it appeared to be generating address book files correctly. The odd thing was in looking at the IIS logs we actually saw quite a few 404 errors of MOC clients trying to request delta files that did not exist. Other users showed successful downloads of the latest delta files which should have included the changes, but they weren’t being applied to their local GalContacts.db for some reason. I also saw those same clients registering a success end up using the fallback logic and downloading older address book files even though they had the newer versions. Very, very strange. Any client we deleted GalContacts.db on would pull down the latest full address book with no issues. The clients looking for deltas that didn’t exist we probably caused by deleting the address book files previously.

Side tip when looking at the IIS logs: Full files start with F-xxxx and delta files follow a D-xxxx-xxxx naming convention. Also, .lsabs files are used by MOC while .dabs files are used by Tanjay devices.

At that point I noticed the mismatch in server (April hotfixes) vs. client (July hotfixes) versions and suggested we get the latest fixes installed on all sides. While that suggestion made its way through change control procedures we opened a PSS case with Microsoft to hit the problem from another angle. The engineer we spoke with immediately blurted out that we needed to match the hotfix versions as soon as we described the behavior. It sounded to me like this was one he had heard before or was familiar with so while we didn’t have a second approach this definitely helped accelerate the change control ticket to an authorized state. After we fully patched the Front-End using the new ServerUpdateInstaller (a lifesaver), applied the back-end database hotfix, and installed the client October hotfix the address book went back to functioning properly. There were a couple of users that needed to delete the GalContacts.db before everything went back to normal, but most of them picked it up without intervention.

As for root cause, the KB 972403 article actually does reference applying both the MOC and server fix together, but the July server hotfix document doesn’t describe this behavior or even mention it. Personally, I think the underlying issue was having the 3.5.6907.37 hotfix on clients while the abserver.exe file was still at 3.5.6907.0. In any case, I learned a lot more about the ABS than I ever cared to, but it was great information that will surely help in the future.

OCS2009-DBUpgrade.msi with a 32-bit SQL Server Back-End

I wanted to point out a quick note about KB 969834 aka the OCS2009-DBUpgrade.msi file – The KB article suggests running the package from your Back-End database server, but if you’re running SQL 2005 x86 you’ll be greeted with the following error:

This installation package is not supported by this processor type.

Basically, the MSI needs to be run from an x64 machine so your only option now is to run the update directly from your Front-End server. If you try to launch from there you might receive this error:

You must install Microsoft SQL Server 2005 Client Tools before you install Microsoft Office Communications Server 2007 R2 (KB969834).

You could try install the SQL Tools and Service Pack updates from installation, but OCS is looking for very specific versions of the SQL tools. The quickest and easiest way is to just use a couple of downloads from the Feature Pack for Microsoft SQL Server 2005 – February 2007.

You’ll want to download and install the following on your R2 Front-End before running the update:

  • Microsoft SQL Server 2005 Backward Compatibility Components (x64 package)
  • Microsoft SQL Server 2005 Management Objects Collection (x64 package)

After running those installers you should be able to run the DB upgrade successfully. Don’t forget – you need to run that MSI from a command line with the poolname (Non-FQDN version) parameter. And if you’re using Server 2008 be sure open the command prompt as Administrator so it runs with elevated rights. Example:

OCS2009-DBUpgrade.msi POOLNAME=MyFirstPool

Your OCS Front-End and DPM 2010 Part 2: Destruction

Now that we’ve verified the DPM backups are running successfully on a regular schedule we can get to really destroying the environment. First up: the RTC database. So shut down your OCS Front-End and SQL services. Then go and delete the RTC.mdf and RTC.ldf files. I know that doesn’t sound like a good idea, but really, delete them.

Open Explorer, jump in to the following volumes and delete the content there:

  • <OCS Installation Volume>\<OCS Installation Folder>\Application Host\Application Data
  • <OCS Installation Volume>\<OCS Installation Folder>\Web Components\AutoUpdate
  • <OCS Installation Volume>\<OCS Installation Folder>\Web Components\Data MCU Web\Web
  • <OCS Installation Volume>\<OCS Installation Folder>\Web Components\Data MCU Web\Non-Web
  • <OCS Installation Volume>\<OCS Installation Folder>\Web Components\DeviceUpdateFiles

Now go and start your SQL services and try starting the OCS services up again. You’ll find a few errors and warnings in your OCS application log because it can’t read the RTC database. Communicator and Live Meeting clients won’t be able to connect to the server as this point either. Oops!

10-9-2009 1-46-19 PM

Congratulations, you’ve successfully messed up your Front-End server to the point where it is non-functional. The device update files have been lost, the MOC Auto-Update files have been lost and all your meeting content is gone. In the next section I’ll demonstrate how to get the server back to an operational state with DPM.

Checking Communicator Endpoint Versions On Your OCS Pools

One of the questions that comes up with OCS deployments that have been around for a while is the question of what clients are connecting to the pool. This can be controlled with client version filters and the auto update feature of R2, but more often than not there are some straggling clients out there. The challenge for those without some sort of configuration management tool is identifying what users have those old clients.

Microsoft has been nice enough to provide a handy tool within the OCS 2007 R2 Management Console that checks what kind of endpoints are connected to your Front-End server. If you open the OCS MMC, click the pool object and then click the database tab you’ll see a number of reports you can run. One of the more useful ones is the client version summary. Just press Go next to it and you’ll see it return a list of endpoints.
clientsummary

You can see from the results we still have quite a mix, and even someone still using a Communicator 2005 client! This is useful in providing an overall picture of what’s been used, but the question I immediately hear next is “Who’s using that version?” Unfortunately, there’s no easy way to tell in the console. You can run a per-user-report which will tell you the endpoints a particular user is signed in with, but that’s going to be a tedious effort to chug through a long list of names trying to find the offenders who haven’t updated their clients. You can see below what searching for a single user returns.
userreport

In order to answer the question of who’s using what we need to run a SQL query against the RTCDyn database. I’ll say this is definitely not a recommended/supported task, so be sure you know what you’re doing here before you start messing around in SQL Management studio. You have the potential to really hose your OCS installation if you start changing database content. The query we’ll run is just a SELECT statement so we shouldn’t cause any problems. Still, you’ve been warned.

Open SQL Management Studio. If you have a standard edition pool you can download and install SQL Management Studio Express for free. Press the New Query button and paste in the following query. Then just press the Execute button. You’ll get a list back of SIP URIs along with the endpoint they are currently using.

SELECT CAST([SipHeaderFrom] as varchar) as "SIP URI"
      ,CAST([ClientApp] AS varchar) as "Endpoint"
FROM [rtcdyn].[dbo].[Endpoint]

That will give us a nice long of everything in use and what SIP URI is signed in with that client.
sql1

Say we want to filter because we’re looking for people with a specific version. In this case, we want to find everyone still using the R1 MOC client so we can add a WHERE clause that searches for strings that match the agent header.

SELECT CAST([SipHeaderFrom] as varchar) as "SIP URI"
      ,CAST([ClientApp] AS varchar) as "Endpoint"
FROM [rtcdyn].[dbo].[Endpoint]
WHERE CAST([ClientApp] as varchar) like '%2.0%'

You could replace that 2.0 with anything else returned in the agent headers such as 3.5, OC, LCC, etc. This only queries the clients that are connected at a specific point in time so you may want to run this from time to time to catch clients that may not have been connected the first time you Hope this helps you identify your clients.

Installing Exchange 2010 in 20 Minutes or Less

This morning I set out to install Exchange 2010 on Server 2008 R2 and I was amazed I actually had this up and running within 20 minutes of booting my guest virtual machine. I have not looked in to many of the technical advantages of R2 over R1 for Exchange yet, but I can say that the installation requires a lot fewer prerequisite installs than on Server 2008 R1. Here’s a quick guide to getting up and running on R2 with all the server roles installed.

  • Install a Server 2008 R2 RTM server. I’d recommend using Enterprise Edition so you can add a 2nd Exchange server later and test out the DAGs. I had a Sysprepped image I was able to boot up and join to the domain very quickly.

  • Copy the Exchange2010-RC1-x64.exe file to your server and run it. Choose a location to extract the files to.

  • Open a command prompt with administrative privileges and navigate to the folder where you extracted the Exchange files.

  • Issue the command: servermanagercmd.exe -ip scripts\exchange-all.xml

  • Ignore the warning about servermanagercmd being deprecated and restart the server when the installation completes.

  • Open the Services MMC.

  • Change the Net.Tcp Port Sharing Service startup type to Automatic. The prerequisite check for the CAS role requires this to be set.

  • Open a command prompt with administrative privileges and navigate to the folder where you extracted the Exchange files.

  • Issue the command: setup

  • Click Choose Exchange language option and then click Install only languages from the DVD.

  • Click Install Microsoft Exchange.

  • Click Next.

  • Accept the license terms and click Next.

  • Select Yes to enable error reporting and press Next.

  • Select Custom Exchange Server Installation and press Next.

  • Select the Mailbox Role, Client Access Role, Hub Transport Role, Unified Messaging Role and Management Tools. Press Next.

  • Name the Exchange organization and press Next.

  • Select No for Outlook 2003 clients or Entourage (pre-Web Services edition) and press Next.

  • Check the box Client Access server role will be Internet-facing, enter your public URL (mail.domain.com) and press Next.

  • Select the option to join the CEIP and press Next.

  • After the prerequisite check completes click Install.

  • You can see my installer completed in about 12 minutes, which is pretty damn cool. This was a VM with 3 GB of RAM with its VHD on a RAID 10 set. Imagine if this was a production machine with a real amount of RAM.

    exchange2010