Lync 2013 SQL Express Instance Memory

Depending on the role, Lync 2013 servers will have one or two SQL Express instances installed as part of the Lync Deployment Wizard steps. All servers deploy an instance called RTCLOCAL, and Lync Front Ends will also deploy an instance called LYNCLOCAL. Generally, there is no configuration or action required on these instances.

If you were to manually install a SQL Express instance the minimum and maximum memory values are essentially unlimited. Technically, this isn't a realistic view because SQL Express is actually limited to 1 GB of memory, but it's important to know the out-of-the-box defaults.

I've noticed that the Lync installation wizard actually configures static minimum and maximum values on each instance of both RTCLOCAL and LYNCLOCAL, and these are based on the amount of RAM presented to a server at installation time. You can see an example here, which is from a FE that uses Hyper-V dynamic memory and only had around 2.19 GB RAM available.
3-12-2014 3-57-38 PM

The instance will always float between 270 MB and 337 MB of memory, which is quite a bit below SQL Express's ability to address up to 1 GB. We want both of these instances to always be able to address 1 GB, so this is not ideal!

Each instance uses slightly different defaults, but for the sake of documentation I've validated the RTCLOCAL instance will be set to a minimum of 12% and maximum of 15% memory. LYNCLOCAL will be set to a minimum of 6% and maximum of 8% memory.

If the server was installed with a minimal amount of RAM, this could negatively impact the Lync services because the SQL instances won't enough memory available to function properly. You'll likely be able to start services, but find they stop after a period of time and log events such as this one:

There is insufficient system memory in resource pool 'internal' to run this query.
Cause: The connection to the database might be broken.

It's fairly obvious that the process needs additional memory, but simply adding memory to the server won't resolve the issue. The "gotcha" here is if you install Lync with minimal memory for some reason and then bump it up later, Lync's SQL Express instances won't actually leverage the memory because of the static max/min values configured at installation time.

You can manually adjust these values within the SQL Express instance, but the Lync process will actually reset your modifications back to these percentages after a period of time.

The permanent fix is to re-run Step 1 and Step 2 in the Lync Deployment Wizard on each machine that had RAM added after the initial installation. This will set the RTCLOCAL and LYNCLOCAL instance max/min memory configuration to the percentages above based on the new amount of installed memory. Now of course, SQL Express still won't be able to access any more than 1 GB of memory, but this fixes any potential ceiling below that number.

Big thanks to my ExtraTeam colleague, Chris Lehr, for helping track this down.

Lync 2013 and the RTCXDS 16 GB Transaction Log Limit

Be aware that when you deploy Lync 2013 there is a maximum size limit that is configured for the transaction log of the RTCXDS database. The other databases created by Install-CsDatabase all have a maximum size set to about 2 TB (so, essentially unlimited), but the RTCXDS log file is strangely capped at 16 GB.

This isn't normally a problem, but if you use SQL Mirroring and/or the Lync Backup Service, which will both leverage this file more, you may start experiencing problems when this limit is reached. KB 2756725 is posted which indicates you may be unable to start the Front End services once this occurs:

Or you may find users are just unable to connect to Lync, and these errors are logged in your Front Ends.

The RtcDb Sync Agent has encountered an Exception: [System.Data.SqlClient.SqlException (0x80131904): The transaction log for database 'rtcxds' is full due to 'LOG_BACKUP'.

If you open the SQL Management Studio you can check the free space available in each log file with the following query:

DBCC SQLPERF (LOGSPACE)

You'll see this information for each database. In this case you can see the RTCXDS database has used 96.95% of the log space available.

full

You can flush this free space out by running a SQL backup job of the transaction logs. I hope it's obvious, but if you've hit the 16 GB limit you're going to create a 16 GB backup file, so make sure you have at least that much free space on your backup target. This is not your usual database-level backup; you'll need to make sure you run a backup of the transaction logs. The specific steps to run this job are outlined here

After the job completes run the query again and notice the change. RTCXDS now has only 0.14% log space used.

empty

So, the question is how to avoid this problem before you get burned. I see two options:

Back Up the Transaction Log File Regularly

Schedule regular backups of the RTCXDS transaction log file. This will flush the file and keep the Log Space Used % at a low value. It's fairly common for Lync deployments to use various scripts for dumping all the backup data to flat files instead of involving SQL admins and backup software, so this may not be an appealing option.

Increase the Transaction Log Maximum Size

Increase the maximum size of the RTCXDS transaction log to match the other databases at 2 TB. You'll most likely run out of disk space on the server before you ever hit this number, and you can catch this issue from cropping up with a basic disk free space alert.

You can follow these steps to adjust the log file maximum size:

  • Connect to the Database Engine of the SQL instance.
  • Expand Databases.
  • Right-click on RTCXDS and choose Properties.
  • Under Select a page, select Files.
  • Select the rtcxds_log file, then move the scroll bar to the right.
  • In the Autogrowth/Maxsize column click the button with the three dots "…"
  • Under the Enable Autogrowth menu, change the Maximum File Size to Unlimited.
  • Click OK twice to commit changes.

Migrating OCS conference directories to Lync the hard way

A few evenings ago I ran into a scenario where moving a conference directory from OCS to Lync failed, and the conference directory ended up in this limbo state where it wasn't on Lync, I couldn't move it back to OCS, and the conferencing attendant wouldn't recognize any PSTNs IDs which were part of the directory. Not a great scenario.

After running Move-CsConferenceDirectory I could verify the move was in progress, but it never completed. The status would show it was trying to move, and OCS eventually started throwing errors that it no longer had a conference directory, but it never fully made it to Lync. The TargetServerIfMoving parameter stayed populated:

Get-CsConferenceDirectory -Identity 5
Identity: 5
ServiceId: UserServer:OCSPOOL.ptown.local
TargetServerIfMoving: UserServer:LYNCPOOL.ptown.local

Trying to run Move-CsConferenceDirectory again would consistently fail with the following errors:

WARNING: Move operation failed for conference directory with ID "5". Cannot perform a rollback because data migration might have already started. Retry the operation.
WARNING: Before using the -Force parameter, ensure that you have exported the conference directory data using DBImpExp.exe and imported the data on the target pool. Refer to the DBImpExp-Readme.htm file for more information.
Exception from HRESULT: 0xC3EE7950, Microsoft.Rtc.Management.ConferenceDirectoryCmdlets.MoveConferenceDirectoryCmdlet

In the end I needed to export the data from the OCS directory via DbImpExp, force the directory to move, and then import the data. Not the cleanest route, but it works. The order is important, so be patient.

On the OCS pool and database export the conference directory data:

DbImpExp.exe /hrxmlfile:C:\Temp\OCSDirectory5.xml /SQLServer:OCS-SQL.ptown.local /restype:confdir

Only once you're positive you have a good export (Read: opened the file and checked!), and made a copy of it you can force the Move-CsConferenceDirectory operation:

Move-CsConferenceDirectory 5 -TargetPool LYNCPOOL.ptown.local -Force

Congrats. You've moved the directory to Lync, but it's empty. Copy the .xml export file to a FE in the Lync pool. On the Lync pool and database import the directory data while specifying the conference directory ID to recover the old data:

DbImpExp.exe /import /hrxmlfile:C:\Temp\OCSDirectory5.xml /SQLServer:LYNC-SQL.ptown.local /restype:confdir /dirid:5

At this point I could see the directory was no longer moving because TargetServerIfMoving was empty, and the conference attendant was now recognizing PSTN IDs which had been created against this directory.

Get-CsConferenceDirectory -Identity 5
Identity: 5
ServiceId: UserServer:LYNCPOOL.ptown.local
TargetServerIfMoving: 

Again, this is a good reason to always do a DbImpExp.exe dump before moving directories or databases around. Those .XML files can save your skin!