Event ID: 30968
Source: Live Communications User Services
Details: The component Live Communications User Services reported a critical error: code C3EE78F8 (Enterprise Edition Server successfully registered with the back-end, but a stored procedure version mismatch was detected. The service will not start until this problem is resolved. Cause: The database schema and the Enterprise Edition Server were updated by different installation packages. Resolution: Ensure both the Enterprise Edition Server and back-end were installed or modified by the same installation package. The service has to stop.

Obviously the error verbiage is a bit outdated with references to LCS, but the error is correct – there is a mismatch between the stored procedure versions which makes the Front-End service to fail to start.

To avoid the issue be sure to apply the latest OCS2009-DBUpgrade.msi package before updating any Front-End servers.

A few months back I was involved in a discussion about what the subject name of an OCS Edge Server’s A/V authentication certificate should be. Some folks were saying to use the Edge server’s internal FQDN and others were saying to use the external, public FQDN you define for A/V. I was in the camp using the external name, but the odd thing was both sides said their approach worked. There is definitely some confusion about what name you should use and Microsoft has actually published directly conflicting information which further confuses the issue. Some testing I’ve recently done clears up why so many documents and people contradict each other – the subject name doesn’t matter. Really. You could put whatever you want in that subject name, assign it to A/V authentication and it will work flawlessly. The purpose of this certificate per the Technet documentation:

The private key of the A/V authentication certificate is used to generate authentication credentials.

Specifically, it’s not used for encryption or MTLS even if that’s not made clear anywhere. Let’s take a step back and clarify a few things for some background:

• There are two services that run on the Edge server with "A/V" in the name. If you’re not familiar with the difference, Jeff Schertz’s More on OCS Edge Server Certificates article has a good explanation for some background on what the difference is between the A/V Authentication and A/V Edge services, but basically – the A/V Authentication service is internal facing and A/V Edge Service is external facing.
• There is no certificate assigned to the A/V Edge service because encryption for external A/V traffic is provided by SRTP.
• The certificate for A/V Authentication is only used by internal clients when trying to communicate with an external or federated client. This means you can (and should) use an internal certificate authority to issue this certificate. There is no benefit or need to use a public certificate for A/V authentication.

Let’s walk through a little example here as if I was trying to figure out what name to use for my A/V authentication certificate. I have the following environment:

• Public Domain: confusedamused.com
• Internal AD Domain: ptown.local
• SIP Domain: confusedamused.com
• Edge Server Internal FQDN: edge.ptown.local
• A/V Edge Service FQDN: av.confusedamused.com

So with that information what should I use as the certificate name for the A/V authentication certificate? If you consult the Technet documentation topic Set up Certificates for A/V Authentication you’ll find this note (emphasis is mine):

The subject name should match the fully qualified domain name (FQDN) of the A/V Edge Service published by the external firewall, or the FQDN of the VIP used by the A/V Edge Service array on the external load balancer (that is, if the Edge Servers are load balanced).

So based on that blurb, my A/V authentication certificate subject name should be av.confusedamused.com. Fair enough.

I ran through the OCS 2007 R2 Edge Planning Tool for a sanity check. You can see the result below, but the tool follows the Technet documentation and uses the external FQDN I defined for the A/V Edge Service when it asked.

A group of MVPs and Microsoft employees published a document called Deploying Certificates in Office Communications Server 2007 which says the following about the A/V authentication certificate (emphasis is mine again):

Must be the FQDN of Audio/Video authentication server in DNS.

Well that calls out the name of the authentication server, not the A/V Edge Service. I think this comes down to really just poor wording in the document which contributes to confusion, but what is the name of our A/V Authentication server? It would be the same name as the internal Edge interface, right? The A/V Authentication server is the Edge server, not the external FQDN. So now we’re being told to use the internal FQDN, edge.ptown.local as the subject name.

Also released by Microsoft was a document called OCS 2007 R2 Walkthrough – Scale to Load Balanced Edge Server which completely contradicts Technet and the Edge Planning Tool (emphasis mine):

• Access Edge Internal (Corporate Certificate). In our sample topology, the subject name would be set to ocsedge.contoso.com, the FQDN of the Edge Server internal interface.
• A/V Authentication Internal (Corporate Certificate). In our sample topology, the subject name would be set to ocsedge.contoso.com, the FQDN of the Edge Server internal interface.

This seems to match up with the certificates document and is somewhat backed by the exact same Technet article I referenced earlier which says:

As a security precaution, you should not use the same certificate for A/V authentication that you use for the internal interface of the Edge Server.

This begs the question "Why would I ever even try to use the same certificate?" The only logical reason would be perhaps because they use the same subject name. That jives with the Scale to a Load Balanced Edge Server documentation. If we’re thinking about this in terms of MTLS connections, you would have to think that this makes the most sense. In your OCS Forest properties if you added an A/V Edge server with the name edge.ptown.local for port 5062, it’s reasonable that you’d expect the A/V Authentication service operating on port 5062 of the internal interface to offer a certificate matching this name. If it presented something wrong, say maybe the external FQDN of the A/V Edge service it should fail, right?

Well, the truth is the name doesn’t matter. There isn’t MTLS validation happening on port 5062 the same way you’d expect MTLS between servers on 5061. I think the reason the certificate requirement issue hasn’t been pointed out yet is because it’s never caused a problem – it works either way. I can use a certificate with a subject name gobblygook.confusedamused.com and media relay authentication through the Edge server works just fine. It just needs a certificate to generate authentication credentials for the media relay process. Go ahead and try it out – put whatever name you want on the certificate and it will still work.

So while the subject name doesn’t really matter, if you’re still interested in adhering to best practices I would recommend using the external facing, public A/V Edge name. In the example earlier this would be av.confusedamused.com. Hopefully Microsoft will update the certificate and scaling documents with a clarification and make them more consistent with the rest of Technet.

Something I’ve been working on lately was a Microsoft case involving inconsistent formatting of numbers. It turns out that MOC actually displays numbers for users in your contact list differently the first time you sign in (i.e. No GalContacts.db exists yet) compared to subsequent sign-ins. This isn’t a normalization problem because the underlying Tel URI is always correct, but actually just a display issue in how the number is presented within MOC.

Apparently the first time you sign in because there is a slight delay in the ABS download (even if you force it immediately) MOC has nothing to go on for contact card information other than the presence XML. If you view the presence XML you’ll see at first it doesn’t actually carry the display format, just the Tel URI, tel:+12345678901 so MOC has to use its own logic to figure out how to display that number. The format it chooses is +1 (234) 567-8901 and there is no way to change that. Not by disabling normalization, using only built-in rules, or by using only company-specific rules – the result is always the same and that display format is hard coded into MOC.

After a lot of back and forth support gave me the ol’ “It’s by design” answer and ended it there. I was a little disappointed because I think MOC should be able to apply the rules immediately after receiving them, but it seems to take another sign-in to take effect. Let me show you what I mean:

Active Directory Fields:

Company Phone Number Normalization file:

Address Book File Dump:

First sign-in uses MOC hard-coded logic:

Subsequent sign-ins display the number as formatted in Active Directory:

Odd, right? Normally this wouldn’t be a problem, but the reason this popped up in the first place was because CUCIMOC was in use and it caches numbers you’ve called previously. So if a user signed in for the first time and dialed another user, CUCIMOC would show you calling with a +1. Even after signing out and back in, CUCIMOC would keep showing the +1 next time you dialed that user because it cached the original number you called, which had the +1. And now any new users you call would not have a +1, creating an ugly inconsistency. We were able to take care of the actual dialing with rules in Call Manager, but it’s just undesirable for end users to see this inconsistency.

Another gotcha I’ll point out is that MOC also tries to respect the access levels here. What I mean by tries is that as we know if a number is in AD it’s visible to everyone in the organization regardless of access level. Say you have a user’s work and mobile numbers in AD and try to view them from another user who’s assigned the company level in MOC, you’ll see MOC apply its own formatting to the work number only. Assign them to the team level and you’ll see MOC also format the mobile number. Bizarre.

Company Access Level on 1st Sign-In:

Team Access Level on 1st Sign-In:

There are two workarounds here since Microsoft refuses to acknowledge this behavior as a bug:

• Format all numbers in AD to the format MOC is going to use on the 1st sign-in. That is, +1 (xxx) xxx-xxxx. Then create a normalization rule in your company file to make sure this gets processed to E.164 by the ABS.
• When a user is signing in to a new PC for the first time (or you had them delete GalContacts.db), have them sign-in once, sign-out after the address book downloads, and finally sign-in again. MOC will now display the numbers from AD instead of formatting them itself. If a user goes to a new PC they need to repeat this process.

Neither of these are great solutions. The first is probably the best, but aren’t we defeating the entire purpose of normalization here? I should be able to put the numbers in any format I want in AD and normalize them with the ABS . Side note before anyone suggests it: This behavior still happens even if you put the AD fields in E.164 (+12345678901) format. For some organizations changing the formatting of the phone field isn’t an option especially if they have some kind of HR software responsible for syncing phone fields, or other applications dependent on the existing formats.

If you want to duplicate the issue yourself, there is a specific use case to make this happen. Most importantly, the user needs to actually be in your contact list.

1. Enter your numbers in AD for User A and B.
2. Ensure the numbers normalize by the ABS.
3. Sign-in as User A.
4. Add User B to your contact list.
5. Sign-in as User B.
6. Add User A to your contact list.
7. Sign-out of both accounts.
8. Delete GalContacts.db and GalContacts.db.idx from both accounts.
9. Sign-in to User A.
10. Sign-in to User B.
11. View User A’s phone numbers from User B’s MOC. You’ll see the MOC internal formatting applied.
12. Sign-out of User B. (Leave User A signed in)
13. Sign-in to User B.
14. View User A’s phone numbers from User B’s MOC. You’ll see the exact format you entered in AD, and for all sign-ins going forward.
15. You can sign-out of User A, delete GalContacts.db again and sign-in to see the MOC formatting again.

Personally, I think the behavior is wrong and needs to be fixed, but Microsoft says otherwise.

The past few days I spent wrestling an address book server issue in OCS and I wanted to share the solution. The quick version: make sure you have your server side and client side hotfix revisions match up.

If you want the whole story…the specific details of this case involved a Front-End server which had the OCS 2007 R2 April hotfixes, but the MOC clients had the July hotfixes applied. The issue first manifested itself with clients reporting ABS damaged:

Communicator cannot synchronize with the corporate address book because the corporate address book file appears to be damaged. Contact your system administrator with this information.

We resolved this by deleting the entire contents of the address book file share and forcing a resync of the address book. We also deleted GalContacts.db from a few user workstations, but later found the client error actually disappeared on its own without removing the file.

Things hummed along nicely for a week or so until a large amount of users (600+) were enabled for OCS one Friday evening. The following Monday previously enabled pilot users were reporting they still didn’t have a SIP URI in their address books for the mass-enabled users. The GalContacts.db file was also still showing a timestamp from the day the mass change occurred, indicating they had not downloaded an update yet.

We took a peek at the Front-End logs and it appeared to be generating address book files correctly. The odd thing was in looking at the IIS logs we actually saw quite a few 404 errors of MOC clients trying to request delta files that did not exist. Other users showed successful downloads of the latest delta files which should have included the changes, but they weren’t being applied to their local GalContacts.db for some reason. I also saw those same clients registering a success end up using the fallback logic and downloading older address book files even though they had the newer versions. Very, very strange. Any client we deleted GalContacts.db on would pull down the latest full address book with no issues. The clients looking for deltas that didn’t exist we probably caused by deleting the address book files previously.

Side tip when looking at the IIS logs: Full files start with F-xxxx and delta files follow a D-xxxx-xxxx naming convention. Also, .lsabs files are used by MOC while .dabs files are used by Tanjay devices.

At that point I noticed the mismatch in server (April hotfixes) vs. client (July hotfixes) versions and suggested we get the latest fixes installed on all sides. While that suggestion made its way through change control procedures we opened a PSS case with Microsoft to hit the problem from another angle. The engineer we spoke with immediately blurted out that we needed to match the hotfix versions as soon as we described the behavior. It sounded to me like this was one he had heard before or was familiar with so while we didn’t have a second approach this definitely helped accelerate the change control ticket to an authorized state. After we fully patched the Front-End using the new ServerUpdateInstaller (a lifesaver), applied the back-end database hotfix, and installed the client October hotfix the address book went back to functioning properly. There were a couple of users that needed to delete the GalContacts.db before everything went back to normal, but most of them picked it up without intervention.

As for root cause, the KB 972403 article actually does reference applying both the MOC and server fix together, but the July server hotfix document doesn’t describe this behavior or even mention it. Personally, I think the underlying issue was having the 3.5.6907.37 hotfix on clients while the abserver.exe file was still at 3.5.6907.0. In any case, I learned a lot more about the ABS than I ever cared to, but it was great information that will surely help in the future.

I wanted to point out a quick note about KB 969834 aka the OCS2009-DBUpgrade.msi file – The KB article suggests running the package from your Back-End database server, but if you’re running SQL 2005 x86 you’ll be greeted with the following error:

This installation package is not supported by this processor type.

Basically, the MSI needs to be run from an x64 machine so your only option now is to run the update directly from your Front-End server. If you try to launch from there you might receive this error:

You must install Microsoft SQL Server 2005 Client Tools before you install Microsoft Office Communications Server 2007 R2 (KB969834).

You could try install the SQL Tools and Service Pack updates from installation, but OCS is looking for very specific versions of the SQL tools. The quickest and easiest way is to just use a couple of downloads from the Feature Pack for Microsoft SQL Server 2005 – February 2007.

You’ll want to download and install the following on your R2 Front-End before running the update:

• Microsoft SQL Server 2005 Backward Compatibility Components (x64 package)
• Microsoft SQL Server 2005 Management Objects Collection (x64 package)

After running those installers you should be able to run the DB upgrade successfully. Don’t forget – you need to run that MSI from a command line with the poolname (Non-FQDN version) parameter. And if you’re using Server 2008 be sure open the command prompt as Administrator so it runs with elevated rights. Example:

OCS2009-DBUpgrade.msi POOLNAME=MyFirstPool

A few weeks ago I started a new job and had to turn in all my UC certified devices to the old employer, which left me needing to pick up some sort of headset for use with Communicator on the road. I took a peek at the Phones and Devices Optimized for Microsoft Office Communicator page and noticed Jabra had a few newly certified devices listed. The Jabra GO 6430 caught my eye mostly because of the  small form factor and sturdy looking design so I decided to give it a shot and placed an order for one.

You can see from the photos below that the device is actually a really nice size.  I’ve had trouble in the past with really small headsets, but I also don’t care for the ones that extend all the way to your mouth. The charging case also doubles nicely as a carrying case, especially for someone who needs to throw a headset in a bag constantly. Unfortunately, the aesthetics are about the only thing Jabra got right. Here are a few photos of the package:

When it arrived I pulled it out, plugged the USB dongle in and tried doing some test calls with Communicator. I placed a call from my mobile to my work number and tried to answer by pressing the multi-function button. It did nothing. Ok, how about outbound? Press the button, and no dial tone. It was as if the button was worthless. Digging a little deeper into the package I found a CD and some documentation (who reads that?) so I popped it in and installed the Jabra Software Suite. After that, I was able to use the multi-function to partially control calls in MOC. For an outbound call I could now get a dial tone by pressing the button, but I still didn’t have much luck with inbound calls. I had some mixed results with the headset either not picking up or it would send the call directly to voicemail, but both were undesirable to say the least.

It only gets worse. Every time I made a settings change within the Jabra suite it seemed take down my entire wireless stack of 802.11 and Bluetooth for a few seconds. At this point I threw in the towel and gave up. Maybe it was my PC, or Windows 7, or some other combination but the bottom line is I shouldn’t have to mess with anything to make these certified devices work flawlessly. I’ve never had issues in the past with any other product, Jabra made or not, but this was unusable. Integrators and especially end-users aren’t going to spend time trying to make these things work – they just expect it to work easily. Giving someone a softphone is already a sensitive subject at times and having a device that flakes out completely ruins any hope of a good user experience. Bottom line: don’t waste your money.

Now would normally be the time where everyone is running around like their head has been cut off because your Front-End server is totally hosed, but because you followed the backup procedures in Part 1 (you did run the backup, right?) restoring service to your OCS server is fairly simple.

Restore the Database

1. Open up the DPM console.
2. Click the Recovery tab at the top.
3. We need to restore the SQL database and files separately, but let’s start with the database. Expand the tree to <Forest Name>\<OCS Server>\All Protected SQL Instances\<OCS Server>\RTC\rtc
   
4. Highlight a suitable recovery date in the calendar and select the RTC database below.
   
5. Right-click and select Recover…
   
6. Press Next.
   
7. We’ve successfully screwed up the server to where we might as well recover to the original SQL server. Select that option and press Next.
   
8. Select Leave database operational and press Next.
   
9. No options needed. Just press Next.
   
10. Yup, those are the files we need. Press Recover.
    
11. Press Close while the recovery operation occurs.
    
12. If you click the Monitoring tab you can view the jobs in process.
    

Restore the Files

1. Now we need to restore files separately. Expand the tree to <Domain Name>\<OCS Server>\All Protected Protected Volumes\<OCS Installation Volume>
2. Highlight a suitable recovery date in the calendar and select the Program Files folder below.
   
3. Right-click Program Files and select Recover…
   
4. Press Next.
   
5. Select Recover to the original location and press Next.
    
6. Select to Overwrite the existing versions (if any), and then select to Apply the security settings of the recovery point version. Press Next.
   
7. Now press Recover.
   
8. Press Close while the recovery operation occurs.
   
9. If you click the Monitoring tab you can view the jobs in process.
   

Fix SQL Database Chaining

One thing DPM won’t restore is an option within SQL. If you miss this step your Front-End services will fail to start.

1. Open SQL Management Studio (Express).
2. Press the New Query button.
3. Enter the following text:

   sp_dboption 'rtc','db chaining',TRUE

4. Press Execute.
   

   <br /><a href="http://www.confusedamused.com/wp-content/pictures/2009/10/image1.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://www.confusedamused.com/wp-content/pictures/2009/10/image-thumb1.png" width="500" height="258" /></a> </li>
   

Bounce the Server

If you check your OCS Front-End you’ll find all the files you deleted previously have now returned. You could probably get away with restarting services as this point, but since the machine was completed hosed I’m just going to restart the server and cross my fingers.

Check Functionality

After the restart all of my OCS services started successfully and my errors have gone away. You can see now my Communicator list still has my contacts and access levels defined. Likewise, Device Updates and client auto updates should function normally now.

Now that we’ve verified the DPM backups are running successfully on a regular schedule we can get to really destroying the environment. First up: the RTC database. So shut down your OCS Front-End and SQL services. Then go and delete the RTC.mdf and RTC.ldf files. I know that doesn’t sound like a good idea, but really, delete them.

Open Explorer, jump in to the following volumes and delete the content there:

• <OCS Installation Volume>\<OCS Installation Folder>\Application Host\Application Data
• <OCS Installation Volume>\<OCS Installation Folder>\Web Components\AutoUpdate
• <OCS Installation Volume>\<OCS Installation Folder>\Web Components\Data MCU Web\Web
• <OCS Installation Volume>\<OCS Installation Folder>\Web Components\Data MCU Web\Non-Web
• <OCS Installation Volume>\<OCS Installation Folder>\Web Components\DeviceUpdateFiles

Now go and start your SQL services and try starting the OCS services up again. You’ll find a few errors and warnings in your OCS application log because it can’t read the RTC database. Communicator and Live Meeting clients won’t be able to connect to the server as this point either. Oops!

Congratulations, you’ve successfully messed up your Front-End server to the point where it is non-functional. The device update files have been lost, the MOC Auto-Update files have been lost and all your meeting content is gone. In the next section I’ll demonstrate how to get the server back to an operational state with DPM.

The goal of this series is to demonstrate how to recover your OCS Front-End’s RTC database in the event of a disaster where your database or disk hosting the RTC database has become corrupted. Or maybe you’ve recovered a server’s installation and configuration, but now need to recover the user information. I’m going to do this in 3 different parts: backing up, destroying, recovering. To get started we need to have a semi-realistic OCS environment running so in this example I have a Standard Edition Front-End running where I’ve done the following:

• Added a few users to my contact list and changed the access levels around.
• Uploaded and approved the latest UCUpdates.cab package for phone devices.
• Added a MOC hotfix to the auto update feature.
• Created a couple of conferences with content in Live Meeting.

These items may seem a little random, but they’ve been done to illustrate what’s restorable from the RTC database and the file shares on a Front-End. I also have another machine called OR1DEVDPM01 running the beta of DPM 2010, which is what we’ll be using for the backup and restore.

Now that we have a machine running we first need to get this thing backed up before we trash it. You’ll want to create an exception on the OCS machine for the firewall to allow any traffic from the DPM machine. This will allow installation of the DPM agent, and allow backups and restores to occur.

Install the DPM Agent

1. Open the DPM console.  
2. Click the Management tab at the top.
    
3. Click the Agents tab below the main navigation line.
4. Click Install in the action pane.
5. Select Install agents and press Next.
   
6. Select the server you’re pushing the agent to (OR1DEVOCS01) and press the Add button. Then press Next.
   
7. Enter the credentials of an account with administrative rights on the server and press Next.
   
8. Since this is a lab and I’m using Server 2008 I’m not too concerned the server restarting, but in production I’d advise opting for the manual restart.
   
9. Press Install and then you can click close while the agent deploys.
   
10. After a minute or two the agent status should change to OK. Now we can start backing up the server.
    

Add the OCS Protection Group

1. In the DPM console again click on the Protection tab.
   
2. Click Create protection group in the action pane.
   
3. Select Servers and press Next.
   
4. Expand the OCS Front-End, OR1DEVOCS01 and you’ll see a few different nodes such as shares, SQL, volumes and system state. The OCS Backup and Restore guide provides some guidance on what actually needs to be backed up from the server. These options pertain to a Standard Edition Front-End so be sure to check the document for any other role. Here are the items we need to select:
   
   • All SQL Servers\<\RTC\rtc<>
   • All volumes\<OCS Installation Volume>\<OCS Installation Folder>\Application Host\Application Data
   • All volumes\<OCS Installation Volume>\<OCS Installation Folder>\Web Components\AutoUpdate
   • All volumes\<OCS Installation Volume>\<OCS Installation Folder>\Web Components\Data MCU Web\Web
   • All volumes\<OCS Installation Volume>\<OCS Installation Folder>\Web Components\Data MCU Web\Non-Web
   • All volumes\<OCS Installation Volume>\<OCS Installation Folder>\Web Components\DeviceUpdateFiles
   
   
5. You can press OK and ignore the warning that pops up about adding the system state backup. Press Next to continue after selecting all of the above options.
   
6. Name the Protection Group something descriptive. I’m going out on a limb here, but I used OCS Front-Ends as the name. I don’t have any tape libraries hooked up, so I’ll just be opting for short-term protection to disk. Press Next. 
   

7. I imagine you’ll generally want more than 5 days of backups, but this works for the purpose here. 15 minute synchronizations are OK, but keep in mind OCS uses a simple recovery model in SQL meaning you take full backups and you restore full backups. None of this full, plus incremental and rolling logs forward fun. Just flat out restore of the entire DB and logs at once. The problem here is a simple database recovery model cannot leverage the synchronization feature of DPM like incremental backups can, so we’re limited to being able to restore only from a full backup, or an “Application recovery point” in DPM terms. You’ll see the default is to back up every day at 8 PM which may or may not be acceptable for you.

   
   

   If you press the Modify button you can add in additional time slots to run an express full backup. Unfortunately (hoping this is a beta bug), you can’t select all the time slots and press Add. So just press Add quite a few times until each timeslot is added and you’ll have a recovery point every 30 minutes for your database. The trade off to running with this kind of frequency is the disk space used. Pick a schedule that’s appropriate for your deployment. Press OK to accept the schedule and then press Next to save the short-term goals.

   
   
8. On the next page you’ll see the disk allocation. Press Next to continue.
   
9. Select when to create the replica of your data (now) and press Next.
   
10. Choose when to run consistency checks and press Next.
    
11. On the last page you can review your selections and then press Create Group. The initial replica jobs will be created and then you can press Close.
    
12. If you click the Monitoring tab you can view the jobs in process.
    

At this point we should have the backups running from the Front-End server. The next part of this will be destroying the data and blowing up the server. After that I’ll show how to recover everything we destroyed.