Lync 2013 CU3 and Hosting Providers

One change I've noticed with the Lync 2013 CU3 (October 2013) Edge server update is how it validates trusted FQDNs found in the topology. Specifically, a FQDN can now either be configured a hosting provider or a federated partner's Access Edge address, but not both.

Under normal circumstances this shouldn't pose an issue, but the Lync Control Panel will not prevent an administrator from performing this configuration. And until CU3, the services didn't seem to care either way.

As an example, the proper way to allow Lync Mobile push notifications for Windows Phone clients (and Lync 2010 iOS clients), has always been to define sipfed.online.lync.com as a hosting provider like in this screenshot.

11-6-2013 9-20-09 PM

The partner push.lync.com should then be defined as a SIP Federated Domain with no Access Edge service specified. But, if an administrator did accidentally specify sipfed.online.lync.com as the Access Edge service FQDN, like in the screenshot here, it had no negative impact on the configuration. 11-6-2013 9-30-54 PM

Until CU3 this didn't present a problem. But after applying CU3 on an Edge server you'll find the Access Edge service will no longer start. Event 14517 will be logged after it fails:

The server configuration validation mechanism detected some serious problems.

ERRORS:
The server at FQDN [sipfed.online.lync.com] is configured as both type 'allowed partner server' and type 'IM service provider' .

The solution here is to make sure any hosting providers, such as Lync Online, are not also defined as the Access Edge address for a SIP federated domain. This issue isn't unique to push.lync.com - it applies to any Office 365 tenants specified as a SIP federated domain.

Make sure you validate your hosting provider and federated domain configurations prior to deploying CU3 on an Edge server.

LYSS.exe High CPU Usage

Not sure on the root cause, but I ran in to an instance where the LYSS.exe process was consuming 80-90% CPU on a Lync 2013 Front End server, and consequently causing issues with conference joins and other Lync functions. This process name is new to Lync 2013 and represents the Lync Storage Service. LYSS runs within an additional SQL Express instance on each Front End server, and is responsible for providing the magic pixie dust that allows Lync to now leverage either SQL or Exchange 2013 Web Services for contacts and archiving data. LYSS provides a layer of abstraction for the internal Lync components to deliver content to, and then sorts out how to deliver it to the appropriate end-state data store. It's essentially a glorified queuing service which replaced the need for MSMQ, so it temporarily stores the data, and then delivers to the appropriate destination (either SQL or Exchange.)

Back to the intent of the post, LYSS doesn't run as its own service so you cannot simply restart it via the Services MMC. If you do run in to this problem, you can kill the lyss.exe process. The service will restart itself automatically, and you'll hopefully see the CPU usage drop.

Lync 2013 and the RTCXDS 16 GB Transaction Log Limit

Be aware that when you deploy Lync 2013 there is a maximum size limit that is configured for the transaction log of the RTCXDS database. The other databases created by Install-CsDatabase all have a maximum size set to about 2 TB (so, essentially unlimited), but the RTCXDS log file is strangely capped at 16 GB.

This isn't normally a problem, but if you use SQL Mirroring and/or the Lync Backup Service, which will both leverage this file more, you may start experiencing problems when this limit is reached. KB 2756725 is posted which indicates you may be unable to start the Front End services once this occurs:

Or you may find users are just unable to connect to Lync, and these errors are logged in your Front Ends.

The RtcDb Sync Agent has encountered an Exception: [System.Data.SqlClient.SqlException (0x80131904): The transaction log for database 'rtcxds' is full due to 'LOG_BACKUP'.

If you open the SQL Management Studio you can check the free space available in each log file with the following query:

DBCC SQLPERF (LOGSPACE)

You'll see this information for each database. In this case you can see the RTCXDS database has used 96.95% of the log space available.

full

You can flush this free space out by running a SQL backup job of the transaction logs. I hope it's obvious, but if you've hit the 16 GB limit you're going to create a 16 GB backup file, so make sure you have at least that much free space on your backup target. This is not your usual database-level backup; you'll need to make sure you run a backup of the transaction logs. The specific steps to run this job are outlined here

After the job completes run the query again and notice the change. RTCXDS now has only 0.14% log space used.

empty

So, the question is how to avoid this problem before you get burned. I see two options:

Back Up the Transaction Log File Regularly

Schedule regular backups of the RTCXDS transaction log file. This will flush the file and keep the Log Space Used % at a low value. It's fairly common for Lync deployments to use various scripts for dumping all the backup data to flat files instead of involving SQL admins and backup software, so this may not be an appealing option.

Increase the Transaction Log Maximum Size

Increase the maximum size of the RTCXDS transaction log to match the other databases at 2 TB. You'll most likely run out of disk space on the server before you ever hit this number, and you can catch this issue from cropping up with a basic disk free space alert.

You can follow these steps to adjust the log file maximum size:

  • Connect to the Database Engine of the SQL instance.
  • Expand Databases.
  • Right-click on RTCXDS and choose Properties.
  • Under Select a page, select Files.
  • Select the rtcxds_log file, then move the scroll bar to the right.
  • In the Autogrowth/Maxsize column click the button with the three dots "…"
  • Under the Enable Autogrowth menu, change the Maximum File Size to Unlimited.
  • Click OK twice to commit changes.