After running Move-CsConferenceDirectory I could verify the move was in progress, but it never completed. The status would show it was trying to move, and OCS eventually started throwing errors that it no longer had a conference directory, but it never fully made it to Lync. The TargetServerIfMoving parameter stayed populated:

Get-CsConferenceDirectory -Identity 5
Identity: 5
ServiceId: UserServer:OCSPOOL.ptown.local
TargetServerIfMoving: UserServer:LYNCPOOL.ptown.local

Trying to run Move-CsConferenceDirectory again would consistently fail with the following errors:

WARNING: Move operation failed for conference directory with ID "5". Cannot perform a rollback because data migration might have already started. Retry the operation.
WARNING: Before using the -Force parameter, ensure that you have exported the conference directory data using DBImpExp.exe and imported the data on the target pool. Refer to the DBImpExp-Readme.htm file for more information.
Exception from HRESULT: 0xC3EE7950, Microsoft.Rtc.Management.ConferenceDirectoryCmdlets.MoveConferenceDirectoryCmdlet

In the end I needed to export the data from the OCS directory via DbImpExp, force the directory to move, and then import the data. Not the cleanest route, but it works. The order is important, so be patient.

On the OCS pool and database export the conference directory data:

DbImpExp.exe /hrxmlfile:C:\Temp\OCSDirectory5.xml /SQLServer:OCS-SQL.ptown.local /restype:confdir

Only once you're positive you have a good export (Read: opened the file and checked!), and made a copy of it you can force the Move-CsConferenceDirectory operation:

Move-CsConferenceDirectory 5 -TargetPool LYNCPOOL.ptown.local -Force

Congrats. You've moved the directory to Lync, but it's empty. Copy the .xml export file to a FE in the Lync pool. On the Lync pool and database import the directory data while specifying the conference directory ID to recover the old data:

DbImpExp.exe /import /hrxmlfile:C:\Temp\OCSDirectory5.xml /SQLServer:LYNC-SQL.ptown.local /restype:confdir /dirid:5

At this point I could see the directory was no longer moving because TargetServerIfMoving was empty, and the conference attendant was now recognizing PSTN IDs which had been created against this directory.

Get-CsConferenceDirectory -Identity 5
Identity: 5
ServiceId: UserServer:LYNCPOOL.ptown.local
TargetServerIfMoving: 

Again, this is a good reason to always do a DbImpExp.exe dump before moving directories or databases around. Those .XML files can save your skin!

We checked the usual suspects: firewalls, permissions, etc with no luck. In the end we just used a different machine that had Topology Builder installed and the same topology file published OK. After doing some more research it seems this is a very generic SQL error that can typically be resolved with a reboot (of the server trying to talk to the SQL instance). So if you hit this error I would suggest just using a different machine, or restart the one you’re using.

Description:
Stopped working

Problem signature:
Problem Event Name: APPCRASH
Application Name: mmc.exe
Application Version: 6.1.7600.16385
Application Timestamp: 4a5bc808
Fault Module Name: KERNELBASE.dll
Fault Module Version: 6.1.7601.17651
Fault Module Timestamp: 4e21213c
Exception Code: e053534f
Exception Offset: 000000000000cacd
OS Version: 6.1.7601.2.1.0.272.7
Locale ID: 1033

In order to troubleshoot errors like this further you can dig in to the log files Topology Builder generates which can be found in the %APPDATA%\Local\Temp folder. I found the real issue by sorting the files in this folder by last modified date which pointed to the Create-CentralMgmtStore log file. Opening this up I noticed the following section:

Installed SQL Server 2005 Backward Compatibility version is 8.05.2312
Connecting to SQL Server on <SQL Server FQDN>
SqlMajorVersion : 10
SqlMinorVersion : 0
SqlBuildNo : 4000
SQL version is acceptable: 10.0.4000.0
Default database data file path is M:
Default database log file path is L:
Error: dbpath must be a fully qualified drive and path, ie c:\sql\db

Notice the final line which points out the database path requires a drive and path. I had been selecting the option to use the SQL database default paths for the CMS and FE pool databases, but these DB paths were simply a drive letter. Since this is considered invalid for Lync I modified the SQL instance default paths to include a folder after the drive letter.

After this change the topology published just fine.

For starters, we followed the steps outlined on Technet. After we had successfully detached and attached all databases and ran the LCSCMD.exe step, we launched the Create Pool wizard and attempted to plug in the info for the new SQL cluster. We got this error back:

An error occurred during the pool backend detection:

Pool backend discovery failed.

Invalid database parameter.

I double-checked the server name, instance, and FQDN and all looked well. We verified the SQL server was accessible via TCP 1433 and no firewall rules were preventing access, so the error didn’t make a lot of sense. Obviously there was some kind of parameter that the wizard GUI was not cool with. I thought maybe this was the SQL allow updates issue, but that solution had no effect on this error. There was definitely some validation check the UI was failing on against our new DB.

Since I couldn’t locate anyone else with this issue I figured my options were to call PSS and extend this process by a few hours, or pull out the ol’ LCSCMD.exe again and try this operation via command line. The Create Pool wizard really is just collecting a bunch of information and then using it to execute the LCSCMD.exe commands in the background so while doing it manually is not fun, it works just as well.

The entire syntax for LCSCMD.exe can be found on Techet, but here is the command we ended up running. Please note, conferencing archiving was not implemented so that paramter is not present.

LCSCMD.exe /Forest /Action:CreatePool /PoolName:MyOCSPool /PoolBE:MySQLServer.ptown.local\OCSInstance /PoolFQDN:MyOCSPool.ptown.local /InternalWebFQDN:MyOCSPool.ptown.local /ExternalWebFQDN:PublicOCSWebComponents.confusedamused.com /RefDomain:ptown.local /ABOutputlocation:\\MyFileServer\AddressBook /MeetingContentPath:\\MyFileServer\MeetingContent /MeetingMetaPath:\\MyFileServer\MeetingMetadata /AppDataLocation:\\MyFileServer\AppData /ClientUpdateLocation:\\MyFilerServer\ClientUpdates /DBDataPath:"D:\Databases" /DBLogPath:"L:\Logs" /DynDataPath:"D:\Databases" /DynLogPath:"L:\Logs" /ABSDataPath:"D:\Databases" /ABSLogPath:"L:\Logs" /ACDDataPath:"D:\Databases" /ACDLogPath:"L:\Logs"

After running the command manually it succeeded with absolutely no issues. The new cluster has been running for over a week now without any issues so I think this is an problem specific to the UI. I’m not sure exactly what causes it, but our environment was running SQL 2008 with SP2 on top of a 2008 R2 SP1 operating system.

As a sidenote, this process seems to undo any changes made by the OCS2009-DBUpgrade.msi patches. You’ll need to re-run the patch version which lines up with your FE patch levels before the FE services will be able to start.

You can export all user data and conference directories with the following command:

dbimpexp.exe /hrxmlfile:everything.xml /sqlserver:SQL.ptown.local\OCS

I usually also grab a separate backup of just the conference directories from the pool:

dbimpexp.exe /hrxmlfile:confdirs.xml /sqlserver:SQL.ptown.local\OCS /restype:confdir

After these run successfully you can copy these files off to a safe place and then proceed with your database operations.

As you are moving the databases around one of the steps on Technet will have you re-run the Create Pool wizard, but if this step fails for any reason the installer will kick into its rollback mode and remove any configuration changes it made. What’s not terribly apparent is that part of this rollback process removes all conference directories on the pool without any warning.

So say this step fails on something silly like a file share permission you’ll suddenly find you dropped all your conference directories. The end result of that is users calling in to meetings via PSTN will no longer be able to enter conference ID and passcode to join the meetings hosted on that pool.

I recently ran a DB move and the user account we used did not explicityly have Full Access rights to one of the OCS file shares (it had been removed at some point for an unknown reason), but the result was the Create Pool operation kicked into rollback mode and removed the pool’s conference directories. We had a solid backup of these to restore from, but this customer had previously lost the directory the first time they tried this operation on their own because of the same problem.

If the directories are dropped and you don’t have a backup via DBImpExp.exe you’ll need to recreate the conference directories on the admin side, but the big pain point is that all users will need to reschedule their meetings (because the previous ID/Passcode mappings are no longer valid). It’s a really ugly user experience and likely to not go over very well. If only you had backed these up in advance!

I would imagine you could restore the conference directory object stored in AD and possibly get that hooked back up to OCS, but your best bet is really to be using DbImpExp.exe instead. A general best practice for any OCS environment is to be taking regular backups of your OCS data and conference directories via DbImpExp.exe so that way you’ll never find yourself in this situation.

If your Create Pool step does fail at least one time you’ll need to restore the directories because they’ve been dumped. After you work out the Create Pool step issue and succeed in starting up your Front-End services you can proceed with the conference diretory restore.

The syntax to restore just the conference directories from the pool is:

dbimpexp.exe /import /hrxmlfile:confdirs.xml /sqlserver:SQL.ptown.local\OCS /restype:confdir

After running this you should be able to dial in via PSTN and enter a conference ID and passcode from a pre-existing meeting again.

This is hopefully a no-brainer, but first you should back up the CMS. Keep in mind that in order to restore the CMS you also need the LIS database, so export both of these.

Export-CsConfiguration –FileName config.zip
Export-CsLisConfiguration –FileName lis.zip

Next, you need to install an empty database to house the new CMS. I typically use the –UseDefaultSQLPaths parameter to control where these databases are installed on the file system. You can also use the -DatabasePaths parameter for more control.

Install-CsDatabase –CentralManagementStore –UseDefaultSQLPaths –SQLServerFQDN <SQL Server FQDN>

Next, enable this topology change:

Enable-CsTopology

Verify all of your replicas are up to date before actually moving the CMS. When ready, log on to a Front-End server for the target pool hosting the CMS (Technet says the source, but this is incorrect. I’ve added a comment there as well). You’ll need to open the Lync Management Shell with elevated privileges (Right-click, Run as Administrator) or the next step will fail.

Move-CsManagementServer

You’ll be prompted to confirm the source and destination. Press Y to accept the move and hopefully it succeeds!

I had to restart both the Master Replica Service and File-Transfer Agent service on the Front-End I used to move the CMS to clear up errors in the event log that it could not connect to the back-end database. Once done, very your replica status is all up to date again.

After the move you need to clean up a few things. You’ll want to launch the setup wizard and run the “Install or Update Lync Server System” step for any Front-End server which was part of the old pool, and for any Front-End server which is part of the new pool. This step adds or removes the required services for a Front-End to serve as a CMS master replica.

Info: Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Cannot create a connection to data source ‘CDRDB’. —> System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 – Could not open a connection to SQL Server)

After searching high and low for a reason why I noticed the data sources for the reports were configured incorrectly by the deployment wizard. Even though the node was part of a SQL cluster the data source was configured as if the SQL server was a standalone local instance. You can see this in the connection string:

To resolve this use the following steps:

1. Browse to the SQL Server Reports home page: http://<Reporting Services Server Name>/Reports.
2. Click on the LyncServerReports folder.
3. Click the Reports_Content folder.
4. You’ll see the CDRDB and QMSDB data sources. Click the CDRDB data source to start.
5. In the connection string field, edit the data source to be the SQL server name (and instance if applicable) instead of the text that says “(local)”.

Repeat these steps for the QMSDB data source. After that, the Monitoring reports dashboard and reports should run. As an example, my CDRDB data source connection string now reads:

data source=LYNCSQL;initial catalog=LcsCDR

When updating the QMSDB the initial catalog parameter is going to be slightly different (QoEMetrics) so be careful to only change the data source if you’re leveraging copy/paste.

It’s not uncommon for Hyper-V hosts to be configured in a failover cluster using dedicated NICs for iSCSI access, but how often does a guest VM typically need access to shared storage? Typically not often. Usually if a VM needs access to the SAN you can connect to the LUN from the host and then pass it through to the guest. With a SQL cluster though, both VMs should reside on separate hosts and need direct access to the SAN. In order to accomplish this you need to initiate iSCSI connections from within the guest VM directly to the SAN.

If you have unlimited NICs this is really easy. Just bind a physical NIC (pNIC) connected to the storage network to a Hyper-V virtual switch, create a new virtual NIC (vNIC) on the Hyper-V guest on this switch, and away you go – the guest now has iSCSI network access. For more realistic scenarios you may be limited on your NIC count and have to get creative because you can’t dedicate a pNIC to VMs only.

If this is true the first option that comes to mind is to leverage VLAN tagging on a virtual switch already using a pNIC. This way you can tag LAN traffic and storage network traffic separately. The downside to this approach is you’re now sharing the same pNIC for both LAN and storage traffic in VMs. The host OS still has a dedicated NIC for iSCSI traffic here. If the LAN traffic starts maxing out the capacity of the adapter you could potentially lose connectivity to the SAN and cause some serious problems for the cluster. This configuration would look like this:

Another option here is to leverage the network adapters already used by the host system for iSCSI access, i.e. the same adapters accessing the SAN and providing high-availability for the guest VMs. That may seem like a poor idea, but I think the alternative of sharing LAN and iSCSI network on the same adapter is far worse. In this configuration the Hyper-V host and the VM guest are both accessing the storage network through the same physical NIC, dedicated purely for storage traffic. A virtual NIC is created on the host and the host traffic also passes through the Hyper-V virtual switch in this scenario. The host and guest sharing a pNIC setup is depicted here:

In order to accomplish this setup you’ll want to use the following steps:

1. Shut down any VMs running on the host OS.
2. Stop any iSCSI access from the host.
3. Create a new Hyper-V virtual switch using the pNIC previously dedicated to iSCSI. Be sure to check the box “Allow management operating system to share this network adapter.”
4. Configure the new network adapter on the host OS with the old pNIC IP address and settings.
5. Reconnect iSCSI targets.
6. Edit the VM guest and add a new network adapter. Assign the network adapter to the iSCSI network virtual switch.
7. Configure iSCSI targets from within the guest VM

At this point, all your storage traffic is isolated to the same physical NIC.

Warning: I cannot for the life of me find documentation from Microsoft on whether or not this actually supported. If I were to wager a guess it’s on the unsupported side, but probably because of the “We haven’t tested this” reason more than the “It doesn’t work” reason. Your mileage may vary on your deployment, but this appears to be working just fine. I was a little reluctant to try this thinking the host OS may have iSCSI performance problems through the Hyper-V virtual switch, but all seems well so far.

I wanted to point out a quick note about KB 969834 aka the OCS2009-DBUpgrade.msi file – The KB article suggests running the package from your Back-End database server, but if you’re running SQL 2005 x86 you’ll be greeted with the following error:

This installation package is not supported by this processor type.

Basically, the MSI needs to be run from an x64 machine so your only option now is to run the update directly from your Front-End server. If you try to launch from there you might receive this error:

You must install Microsoft SQL Server 2005 Client Tools before you install Microsoft Office Communications Server 2007 R2 (KB969834).

You could try install the SQL Tools and Service Pack updates from installation, but OCS is looking for very specific versions of the SQL tools. The quickest and easiest way is to just use a couple of downloads from the Feature Pack for Microsoft SQL Server 2005 – February 2007.

You’ll want to download and install the following on your R2 Front-End before running the update:

• Microsoft SQL Server 2005 Backward Compatibility Components (x64 package)
• Microsoft SQL Server 2005 Management Objects Collection (x64 package)

After running those installers you should be able to run the DB upgrade successfully. Don’t forget – you need to run that MSI from a command line with the poolname (Non-FQDN version) parameter. And if you’re using Server 2008 be sure open the command prompt as Administrator so it runs with elevated rights. Example:

OCS2009-DBUpgrade.msi POOLNAME=MyFirstPool

Now would normally be the time where everyone is running around like their head has been cut off because your Front-End server is totally hosed, but because you followed the backup procedures in Part 1 (you did run the backup, right?) restoring service to your OCS server is fairly simple.

Restore the Database

1. Open up the DPM console.
2. Click the Recovery tab at the top.
3. We need to restore the SQL database and files separately, but let’s start with the database. Expand the tree to <Forest Name>\<OCS Server>\All Protected SQL Instances\<OCS Server>\RTC\rtc
   
4. Highlight a suitable recovery date in the calendar and select the RTC database below.
   
5. Right-click and select Recover…
   
6. Press Next.
   
7. We’ve successfully screwed up the server to where we might as well recover to the original SQL server. Select that option and press Next.
   
8. Select Leave database operational and press Next.
   
9. No options needed. Just press Next.
   
10. Yup, those are the files we need. Press Recover.
    
11. Press Close while the recovery operation occurs.
    
12. If you click the Monitoring tab you can view the jobs in process.
    

Restore the Files

1. Now we need to restore files separately. Expand the tree to <Domain Name>\<OCS Server>\All Protected Protected Volumes\<OCS Installation Volume>
2. Highlight a suitable recovery date in the calendar and select the Program Files folder below.
   
3. Right-click Program Files and select Recover…
   
4. Press Next.
   
5. Select Recover to the original location and press Next.
    
6. Select to Overwrite the existing versions (if any), and then select to Apply the security settings of the recovery point version. Press Next.
   
7. Now press Recover.
   
8. Press Close while the recovery operation occurs.
   
9. If you click the Monitoring tab you can view the jobs in process.
   

Fix SQL Database Chaining

One thing DPM won’t restore is an option within SQL. If you miss this step your Front-End services will fail to start.

1. Open SQL Management Studio (Express).
2. Press the New Query button.
3. Enter the following text:

   sp_dboption 'rtc','db chaining',TRUE

4. Press Execute.
   

   <br /><a href="http://www.confusedamused.com/wp-content/pictures/2009/10/image1.png"><img style="border-right-width: 0px; display: inline; border-top-width: 0px; border-bottom-width: 0px; border-left-width: 0px" title="image" border="0" alt="image" src="http://www.confusedamused.com/wp-content/pictures/2009/10/image-thumb1.png" width="500" height="258" /></a> </li>
   

Bounce the Server

If you check your OCS Front-End you’ll find all the files you deleted previously have now returned. You could probably get away with restarting services as this point, but since the machine was completed hosed I’m just going to restart the server and cross my fingers.

Check Functionality

After the restart all of my OCS services started successfully and my errors have gone away. You can see now my Communicator list still has my contacts and access levels defined. Likewise, Device Updates and client auto updates should function normally now.